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ABSTRACT 



The effects of adult education in welfare- to-work programs 
were examined in a national evaluation of welf are-to-work strategies. The 
evaluation used a random research design to estimate the overall effects of 
welfare-to-work programs in the following states: Georgia, Ohio, Michigan, 
Oklahoma, Oregon, and California. The study focused on the following issues: 
the quality of the education services provided; the extent to which welfare 
recipients participate in education and earn education credentials; the value 
of the education services provided; and the value of basic skills and 
education credentials in the labor market during the mid-1990s. The following 
were among the key findings: (1) even when welfare recipients preferred not 

to enter adult education, welfare-to-work programs substantially increased 
their receipt of such education; (2) assignment to education- focused programs 
did not generally appear to have substantial payoffs for welfare recipients 
in terms of education outcomes; and (3) earning a General Educational 
Development (GED) certificate, increasing basic skills, or subsequently 
participating in postsecondary programs yielded substantial benefits in terms 
of employment, earnings, and self-sufficiency. The following items are 
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appended: discussions of the data sources and research samples; five 
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Chapter 1 



Admit Education m Mandatory Welfare-to-Work Programs” 
Introduction, Synthesis, and Implications for Policy 



I. Introduction! 



Since the early 1980s, welfare policymakers and program operators have debated the role 
of adult education in program strategies to help welfare recipients make the transition from wel- 
fare to work. The so-called Human Capital Development, or HCD, strategy focuses on providing 
education and increasing welfare recipients’ basic academic skills and education credentials, fol- 
lowing research evidence that these skills and credentials are prerequisites to obtaining stable 
employment. 1 This HCD strategy became popular especially during the mid to late 1980s. For 
example, the California Greater Avenues for Independence (GAIN) program, initiated in 1986 
and evaluated by Manpower Demonstration Research Corporation (MDRC), provided education 
and training to large numbers of welfare recipients. During the same time, the federal govern- 
ment also placed a greater emphasis on adult education, as evidenced in the Family Support Act 
(FSA) of 1988, Public Law 100-485. 

However, during the early 1990s, alternative program strategies gained popularity, seek- 
ing rapid job entry for welfare recipients instead of providing them with education first. Such 
strategies — known as Labor Force Attachment (LFA), or “work first,” strategies — are supported 
by research findings suggesting that quickly entering the labor force is a promising trajectory to 
long-term self-sufficiency. 2 For example, in 1995, California ended testing of literacy and math 
skills as welfare recipients enter the state’s welfare-to-work program, thereby eliminating these 
tests as a way to determine who initially needs education services. This turn toward more work- 
focused welfare-to-work programs was reinforced by the Personal Responsibility and Work Op- 
portunity Reconciliation Act (PRWORA) of 1996, P. L. 104-193, which placed time limits on 
welfare receipt, making longer stays in education and training programs less attractive for pro- 
gram operators and welfare recipients. 

However, the debate surrounding HCD programs for welfare recipients is not settled. Al- 
though LFA programs appear to be more effective than education-focused programs in the short 
term, there is no proof that offering job search programs to all welfare recipients (regardless of 
their education needs) leads to long-term self-sufficiency for a majority of the welfare caseload. 
A significant number of programs currently operated for welfare recipients offer education 
classes among their array of possible services, and such services are likely to remain important 
under Temporary Assistance for Needy Families (TANF), the funding structure for PRWORA. 

Comparisons of HCD and LFA strategies are not a major focus of this report. Instead, this 
report mostly addresses key questions about how HCD programs in general, and adult education 
activities in particular, affect the educational and economic outcomes of welfare recipients. Spe- 



'Mincer, 1974; Polachek and Siebert, 1993; Sum et al., 1995. 
2 See, for example, Riccio et al., 1994. 
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cifically, these analyses move beyond overall HCD program effects and focus on participation in 
adult education and the effects of such participation. This is important, because many welfare 
recipients who are assigned to HCD programs do not enroll in adult education classes or they 
drop out of these classes after only a brief spell of participation. As a result, the impacts of the 
larger programs do not necessarily reflect the full potential of the adult education services pro- 
vided to those who receive them. Learning more about how those specific services affect partici- 
pants is the primary focus of this report. 



Background Information: Research Design 
for the National Evaluation of Welfare-to-Work Strategies 

This report is part of the National Evaluation of Welfare-to-Work Strategies (NEWWS) — an 
evaluation of programs begun under the Family Support Act, conducted by the Manpower Dem- 
onstration Research Corporation, and funded by the U.S. Department of Health and Human Ser- 
vices, with support from the U.S. Department of Education. The evaluation includes programs in 
seven sites across the country: Atlanta, Georgia; Columbus, Ohio; Detroit, Michigan; Grand 
Rapids, Michigan; Oklahoma City, Oklahoma; Portland, Oregon; and Riverside, California. It 
uses a random assignment research design to estimate the overall effects of the welfare-to-work 
programs being studied. 

In each site, individuals who were required to participate in the program were assigned, 
by chance, either to a program group that had access to education, training, and other employ- 
ment services and whose members were required to participate in the program or risk a reduction 
in their monthly welfare grant (a “sanction”), or to a control group, which received no services 
through the program but whose members could seek out services from the community. This ran- 
dom assignment design ensures that there were no systematic differences between the back- 
ground characteristics of people in the program group and those in the control group when they 
entered the study. Thus, any subsequent differences in outcomes between the groups can be at- 
tributed to the program. 

It is important to note that the programs in the evaluation were not subject to the condi- 
tions and requirements of what are currently referred to as “welfare-to-work” programs as de- 
fined under TANF. During the follow-up period analyzed in the report, individuals in the 1 1 
studied programs did not face a time limit on eligibility for welfare assistance as they would now 
under TANF. All the programs, however, shared TANF’s primary goal of moving welfare recipi- 
ents into paid work and off assistance. Furthermore, among the 1 1 programs are some which are 
strongly employment-focused — the welfare-to-work strategy favored under TANF — as well as 
some which are strongly basic education-focused. The programs varied in many other ways as 
well, including how broadly the participation mandate was applied to the welfare caseload and 
how strictly it was enforced, the amount of child care support provided for program participation 
or employment, and methods of case management. The programs also served different welfare 
populations and operated in a variety of labor markets. Lastly, it is important to point out that the 
programs being evaluated here are state-operated welfare-to-work programs originally developed 
under the Family Support Act of 1988. These programs are unrelated to the “welfare-to-work” 
programs currently being operated by local Workforce Investment Boards, supported by the 
Workforce Investment Act, and administered by the U.S. Department of Labor. 

The research design for the study presented in this report often diverges from that of 
the larger NEWWS study. Instead of comparing randomly assigned program group members 
and control group members, this study often compared the experiences of welfare recipients 
who participated in certain adult education activities with those of recipients who did not and 
sought to estimate how varying degrees of participation affected education and employment 
outcomes. A detailed discussion of these “nonexperimental” comparisons is featured in a sepa- 
rate text box on page 7. 
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A. Purpose off Titans Study 

The analyses presented here help answer many important policy questions surrounding 
adult education for welfare recipients. These questions concern (1) the quality of the education 
services provided, (2) the extent to which welfare recipients participate in education, (3) the ex- 
tent to which welfare recipients earn education credentials, (4) the value of the education services 
provided, and (5) the value of basic skills and education credentials in the labor market during 
the mid-1990s. These issues are addressed primarily by comparing the experiences of recipients 
who participate in adult education with those of recipients who do not and by assessing the rela- 
tive effectiveness of different levels of participation in adult education. 

B. Overview of Th 5s Report 

These analyses of how adult education works in the context of welfare-to-work programs 
were conducted for a large sample of welfare recipients who entered one of the 1 1 programs 
studied in the National Evaluation of Welfare-to-Work Strategies (NEWWS) without a high 
school diploma or General Educational Development (GED) certificate. 3 Thus, the findings do 
not generalize to welfare recipients who do have a high school diploma but who still may be 
served by HCD programs that provide more advanced levels of education and training. The types 
of adult education examined in the report encompass adult basic education (ABE) classes, pro- 
grams preparing students for the GED exam, regular high school classes, and classes in English 
as a Second Language (ESL). Among these, ABE and GED preparation accounted for most of 
the adult education in the 1 1 mandatory welfare-to-work programs studied. These 1 1 programs 
operated in seven sites, and each program was operated under the federal FSA and its Job Oppor- 
tunities and Basic Skills Training (JOBS) program. (Program intake for this study began in June 
1991 and ended in December 1994; data presented cover June 1991 through December 1997. See 
the accompanying box for further information about this study and report.) 

This chapter summarizes most, but not all, 4 of the analyses presented as a collection of 
papers in this report, specifically addressing the following questions: 

o What are the characteristics of adult education providers in welfare-to-work 
programs? What are typical attendance patterns in these classes? 

o To what extent, and for whom, do welfare-to-work programs increase partici- 
pation in adult education services and increase educational attainment and 
achievement? 

o Do education-focused welfare-to-work programs improve education out- 
comes? 

o What is the payoff to additional participation in adult education? 



3 GED refers to the GED credential and the exams that individuals must pass to attain it. In this report, we use 
GED to refer to both the credential and the exams. 

4 For example, Chapter 5 of this report presents comparisons of program impacts on earnings and welfare receipt 
for welfare recipients in LFA and HCD programs. Because these comparisons do not directly relate to participation 
in adult education and outcomes directly associated with such participation, these analyses are not summarized here. 
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° How do education outcomes and milestones affect the employment outcomes 
and self-sufficiency of welfare recipients? 

° Among those who participate in adult education, who moves on from adult 
education to receive postsecondary education and training, and how does this 
contribute to their earnings and self-sufficiency? 

The analyses in the report take a unique perspective on adult education, one that will be 
of interest to the adult education community as well as to those involved in welfare policy. First, 
our analyses reflect on the effectiveness of adult education services provided to a highly disad- 
vantaged group of students: low-income, mostly jobless, single-parent women who lack a high 
school diploma or GED and are receiving welfare. This group represents a significant share of all 
adult education students. One study found that, in 1992, 22 percent of all new students in U.S. 
adult basic education, high school completion, and GED programs had received public assistance 
in the year before enrollment; about 1 1 percent of all new ESL students met this criterion. 5 This 
same disadvantaged group is likely to be of increasing concern to welfare policymakers. Drastic 
reductions in welfare caseloads since their peak in 1994 are also changing the face of caseloads, 
which now increasingly consist of “hard-to-serve” recipients. It is likely that many of those left 
on the welfare rolls will lack an education credential and will have poor reading or math skills. 6 

Second, the adult education programs studied and their effects on students reflect the fact 
that these programs operate within the context of welfare-to-work programs. Such programs pro- 
vide other services, such as counseling, child care, job search assistance, and postsecondary edu- 
cation and training. Although many issues facing adult educators are essentially unrelated to the 
welfare status of the students they serve, the context of available supports, expectations, and re- 
quirements is different for those enrolled in adult education as part of a welfare-to-work program. 
Thus, the measured effects of these programs reflect not only the payoff to adult education but 
the effects of a larger package of services and requirements that included adult education. As part 
of such a package, the adult education provided could be strengthened to produce greater effects 
(for example, if students receive help with child care or transportation). However, the effects of 
adult education also could be weakened by other program components (for example, if program 
rules limit the time that students may be enrolled in adult education or if the program emphasizes 
a quick transition from welfare to work). 

Third, for welfare recipients in our study, participation in adult education was mandatory. 
While “traditional” adult education students enroll on a voluntary basis and can therefore be pre- 
sumed to be motivated to learn, such motivation may sometimes be lacking when students are 
compelled to participate by mandatory welfare-to-work programs. Like most other adult educa- 
tion students, those mandated to participate often have done poorly in school in the past and may 
be alienated from traditional educational institutions and modes of instruction. Unlike the volun- 
tary, or traditional, students, however, students connected to a welfare-to-work program may ini- 
tially be motivated to attend classes less by the desire to learn or to obtain a credential than by the 



5 Young et al., 1994, p. 15. 
6 Danziger et al., 1999. 
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Deffmittioim off Key Terms 



Many of the terms used in this report are not reintroduced and redefined in each chapter. In some 
cases, these terms are ambiguous or have a somewhat different meaning in the context of wel- 
fare-to-work programs than is common in the adult education field. In this box, we introduce 
some of the most commonly used terms in this report. 

Throughout the report, adult education refers to any or all of the following: 

o Adult basic education (ABE) classes. These provide reading and math instruc- 
tion to students whose achievement levels are lower than is required for high 
school completion or GED classes — typically at the eighth grade level or lower. 

O General Educational Development (GED) classes. These prepare students to 
take the GED tests that are used by states to award certificates signifying knowl- 
edge of basic high school subjects (social studies, literature, science, math, and 
writing). Students entering GED classes usually are expected to have language and 
math skills at a ninth grade level or higher so that they can use the instructional 
materials. 

® High school completion classes. These replicate a high school curriculum in an 
adult school setting. Students usually must have language and math skills at a 
ninth grade level or higher to enter a high school completion program. When stu- 
dents finish the course of study, they receive a high school diploma. 

e English as a Second Language (ESL) classes. These provide instruction in how 
to read and write English to people who are not fluent English speakers. 

The term welfare encompasses both Temporary Assistance for Needy Families (TANF) and its 
predecessor, Aid to Families with Dependent Children (AFDC). 

Program group members are sample members who were randomly assigned to a welfare-to-work 
program. 

Control group members are sample members who were randomly assigned to a control group, 
were excused from a welfare-to-work program mandate, and were ineligible to receive most pro- 
gram services. However, control group members could seek out similar services in the commu- 
nity on their own. 

Program participants are sample members who participated in a particular program activity. 
This could be a class, a training program, a job club, or something similar. Program participation 
is not limited to program group members assigned to welfare-to-work programs, because control 
group members could access services outside welfare-to-work programs on their own. Depending 
on the context, program participants are sometimes referred to as “students,” as “enrollees,” or as 
“recipients” of education or training services. 
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need to comply with welfare-to-work program requirements in order to avoid reductions in their 
welfare grant. 

Finally, in the context of welfare-to-work programs, adult education is viewed as an in- 
termediate goal, not as an end in itself. Financial self-sufficiency of adult students and their fami- 
lies is the ultimate goal of these programs. 

These factors have helped shape the analyses and interpretation of results in this report. 
However, one additional factor — one of importance to the adult education community — cannot 
be taken into account in this examination. The prevalence of learning disabilities among welfare 
recipients is estimated to be between 25 and 50 percent. 7 As implemented during the study period 
and probably continuing into current operations, most programs did not assess welfare recipients 
for learning disabilities, which could affect the programs’ ability to address these disabilities, cli- 
ents’ skill development in the programs, and clients’ subsequent labor market success. 

C. Findings in Brief 

Although the five chapters following this one use various analytical techniques and sam- 
ples, taken as a whole they support the following broad conclusions about adult education in the 
NEWWS welfare-to-work programs serving those without a high school diploma or GED: 

° 1° providing services for welfare recipients, adult education programs gener- 

ally did not adapt their curricula or teaching methods to fit the specific needs 
of this group of students. 

° Even when welfare recipients preferred not to enter adult education, welfare- 
to-work programs substantially increased their receipt of such education. 

There was no evidence that those who were mandated to participate (most of 
whom did not express a preference for adult education) benefited any less 
from their participation in terms of educational attainment and literacy or math 
gains than those who volunteered. 

® On the whole, assignment to education-focused programs did not appear to 
have a substantial payoff for the welfare recipients in our study in terms of 
their education outcomes. Although the programs increased GED receipt, 
most participants did not earn a GED, and few experienced significant in- 
creases in their reading and math skills. Three-year impacts on earnings and 
welfare receipt in HCD programs were smaller than those experienced by wel- 
fare recipients in LFA programs. 



7 See, for example, Center for Law and Social Policy, 1998, and Pavetti, 1997. 




The random assignment research design (described in the box on page 2) was used for cer- 
tain analyses in this report, but others were conducted with nonexperimental methods, 
which go beyond the random assignment research design. In general, the distinction be- 
tween these two methodologies depends on whether a question concerns effects of the 
programs as a whole, which the random assignment research design is well equipped to 
address, or whether a question concerns the effects of program components or program 
outcomes, such as participation in adult education, GED receipt, or participation in post- 
secondary services. Because of the protection offered by the random assignment research 
design, findings about the programs as a whole (concerning questions like “By how much 
did the HCD programs increase participation in adult education?”) are more reliable and 
can be presented with greater confidence. Findings about program components or out- 
comes (concerning questions like “By how much does a GED increase subsequent enroll- 
ment in postsecondary education or training?”) are not protected by random assignment 
and therefore have greater uncertainty surrounding them. 

In addition to the difference in research methods, the two types of questions out- 
lined here differ in their substantive focus. The “program” questions are less precise than 
the “component” or “outcome” questions, describing how assignment to a broad program 
affected outcomes, not how specific events and services did. For example, many of those 
assigned to welfare-to-work programs did not participate in program activities, or partici- 
pated for short periods. This limits the extent to which the program could affect sample 
members’ education outcomes. No such limitations exist for analyses involving specific 
program components or outcomes because participation is explicit in the definition of the 
measure studied. An estimate of the effects of GED receipt on earnings in principle (and 
on average) applies fully to everyone who received such a credential, as does an estimate 
of the payoff to an additional month of adult education. Thus, in summary, the experimen- 
tal findings presented are robust and reliable but apply to programs that do not always 
reach participants as intended, whereas the nonexperimental findings are less reliable ana- 
lytically, but answer more concrete policy questions facing the adult education commu- 
nity. 



• Gains in reading skills appeared to vary with the length of time spent in the 
adult education programs. Stays shorter than a year (which the majority of par- 
ticipants in adult education had) did not improve reading skills measurably, 
whereas longer stays were associated with substantial gains, comparable tor 
this sample — to those associated with regular high school attendance. 

• Improvements in math skills were associated with shorter spells of adult edu- 
cation. After six months of adult education, most participants math skills no 
longer improved. 

• GED receipt also was associated with shorter spells of participation in adult 
education. Additional participation beyond six months did not increase GED 
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receipt, possibly because most GED recipients were close to being able to pass 
a GED test when they entered the programs. 

® Higher average levels of teachers’ experience and education in the adult edu- 
cation programs appeared to enhance the payoff to participation in adult edu- 
cation in terms of reading and math skills. 

• The welfare recipients who were most likely to get GED certificates and re- 
ceive postsecondary services were those who had higher initial reading and 
math skills when they entered the welfare-to-work programs. 

® As students earned GEDs, increased basic skills, or subsequently participated 
in postsecondary programs, they appeared to have substantial benefits in terms 
of employment, earnings, and self-sufficiency. However, relatively few adult 
education participants received a GED, increased their basic skills, or entered 
postsecondary programs. 

® Receipt of a GED credential was an important predictor of subsequent enroll- 
ment in postsecondary programs. Participants in basic education programs 
who went on to postsecondary education or training programs appeared to ex- 
perience substantial benefits from them in terms of increased earnings and 
self-sufficiency. 



II. Summary of Each Chapter’s Findings 

This report on adult education in 1 1 welfare-to-work programs addresses many different 
aspects of the adult education experience of welfare recipients in welfare-to-work programs. 
Specifically, the report traces the steps outlined in Figure 1.1, 8 which describes the hypothesized 
effects of adult education in welfare-to-work programs. 

In Chapter 2 we focus on the second and third boxes in Figure 1.1, addressing the ques- 
tions of how education-focused welfare-to-work programs affect participation in adult education, 
what adult education that is provided to welfare recipients looks like, and how these welfare-to- 
work programs affect education outcomes. Specifically, program effects are presented on partici- 
pation in adult education, basic skills, and GED receipt in three education-focused programs (in 
Atlanta, Grand Rapids, and Riverside). We explore how these effects vary across different sub- 
groups and attempt conceptually to link effects on one outcome to effects on other outcomes. 

In Chapter 3 this analysis is taken a step further, focusing more directly on the third box 
in Figure 1.1, describing the relationship between participation in adult education and improve- 
ments in the skill and educational attainment of participants. For example, we examine how an 
extra month of participation affects key education outcomes and how this effect varies with (1) 
total time spent in adult education, (2) individual student characteristics, and (3) program and 
staff characteristics. 



8 Adapted from Martinson and Friedlander, 1994. 
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National Evaluation of Weflfare-to-Work Strategies 



Figure 1.1 

For Sample Members Without a High SchooB Diploma or GED at Random Assignment: 
Hypothesized Effects off Adult Education in Welfare-to-Work Programs 
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In Chapter 4 we look at the fourth box and link education outcomes to employment out- 
comes and self-sufficiency, addressing questions like “What is the GED worth to welfare recipi- 
ents?” and “How are additional reading and math skills valued in the labor market?” Chapter 4 
uses survey and administrative data across all 1 1 welfare-to-work programs in each of the seven 
NEWWS sites. 

Chapter 5 revisits the experimental comparisons (of randomly created program and con- 
trol groups), comparing all 1 1 programs for their effects on earnings and welfare receipt and at- 
tempting to isolate factors that made some programs more successful than others. 

Last, Chapter 6 completes our analyses by focusing on the box to the right in Figure 1.1, 
addressing the important intermediate step of postsecondary education and training, which is of- 
ten believed to be an important intermediary link between participation in adult education and 
longer-term improvements in earnings and other employment outcomes. The chapter addresses 
the questions: “What determines whether adult education participants enter postsecondary pro- 
grams?” and “Do participants benefit from these programs?” 

A. Characteristics of Adult Education in Welfare-to-Work Programs and the 
Effects of Education-Focused Welfare-to-Work Programs on Edncationall 
Attainment and Achievement (Chapter 2) 

Site visits and surveys of education providers in the HCD programs in Atlanta, Grand 
Rapids, and Riverside found that these welfare-to-work programs used a wide variety of educa- 
tional institutions to provide adult education to the welfare recipients enrolled in these programs. 
Research conducted at the adult education provider sites concluded that the inclusion of welfare- 
to-work program participants in the adult education classes usually did not greatly affect the pro- 
viders’ operations, curricula, and teaching methods. In other words, welfare-to-work program 
participants took classes together with non-welfare recipient adult education students and gener- 
ally did not receive services specially tailored to their needs from adult education providers or 
classroom teachers. In some cases, hours were expanded to enable welfare recipients to partici- 
pate for 20 hours a week, as required by welfare-to-work program regulations. In other cases, ad- 
ditional counseling or job-readiness instruction were added for welfare recipients. Only in River- 
side did the welfare-to-work program negotiate contracts with adult education providers and use 
welfare-to-work funds to pay providers serving their clients. However, aside from these contrac- 
tual differences, there were few systematic differences in the adult education provided across the 
three sites. 

The welfare-to-work programs substantially increased participation in adult education for 
those who entered the study without a high school diploma or GED (the sample analyzed 
throughout the report). Without the programs, about one-fifth of sample members sought out 
adult education programs on their own (as evidenced by two-year participation rates in the con- 
trol group). The three programs studied in Chapter 2 (Atlanta, Grand Rapids, and Riverside) 
more than doubled this rate of participation: one-half of program group members participated in 
adult education. When they enrolled in adult education, program group members also stayed 
longer. Across all sample members (including those who did not participate at all), the program 
more than tripled the average number of hours of adult education, from 68 for control group 
members to 244 for program group members. This means that the average participant in pro- 
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gram-provided adult education classes was enrolled for about 488 hours. Thus, the programs in- 
duced more individuals to participate in adult education, and those who participated did so for 
more hours. 

As part of the analysis, impacts on adult education participation were estimated for 20 
different subgroups, defined using individual characteristics measured at program entry (hereafter 
referred to as “baseline characteristics”). 9 Examples of such subgroups include persons with 
young children, those who dropped out of school having completed eighth grade or less, those 
expressing a lack of desire to go back to school, and those with personal or family barriers to par- 
ticipation. Without exception, the three HCD programs increased participation in adult education 
for each of these subgroups. This shows that mandatory welfare-to-work programs can increase 
welfare recipients’ exposure to adult education even among welfare recipients with barriers to 
participation. 

The three programs achieved modest impacts on GED receipt during a two-year follow- 
up period. Whereas only 4 percent of control group members received a high school diploma or 
GED during the follow-up period, 1 1 percent of program group members received such a creden- 
tial. This impact more than doubled the proportion with an education credential; however, fewer 
than one in five participants in adult education earned a credential. (Many sample members 
might not be expected to attain such a credential during the two-year follow-up period, because 
they entered the programs with low achievement levels or limited English skills.) 

Moreover, the three programs did not increase scores on standardized reading and math 
tests, conducted as part of the two-year follow-up interview. As discussed more extensively in 
Chapter 2, the combination of modest increases in GED receipt and a lack of gains in measured 
literacy and math skills has been found in several previous studies. There are several possible 
explanations for this apparent discrepancy. First, it is possible that the GED test and the basic 
literacy and math tests administered in the survey do not capture the same underlying skills. In 
that case, someone might be able to pass a GED test without showing concomitant gains in basic 
reading and math skills. It also is possible that difficulties with the administration of the literacy 
and math tests reduced the statistical reliability of our findings. (These tests were administered as 
part of a long interview in sample members’ homes — not the ideal environment in which to con- 
centrate on a skills test.) However, it is reasonable to conclude that, on average, participants ex- 
perienced limited benefits in terms of increased skills and credentials from their participation in 
adult education. Subsequent analyses presented in Chapter 3 and 4 further explore this issue. 

Impacts on GED receipt and educational achievement also were estimated separately for 
20 subgroups. Researchers found that impacts on GED receipt were strongest for those entering 
the welfare-to-work programs with already high reading and math skill levels. Those entering 
with high reading scores experienced an impact (that is, an increase relative to the control group) 
of 16 percentage points. Conversely, those entering with low reading scores experienced an im- 
pact of only 3 percentage points. Those entering the program having left school below the ninth 



9 When subgroups are defined using preprogram baseline characteristics, comparisons of outcomes for program 
and control groups within these subgroups are considered fully “experimental”; that is, these comparisons constitute 
unbiased estimates of the program effects for those particular subgroups. 
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grade experienced no impact on GED receipt. Thus, it seems that the programs’ effects on GED 
receipt were closely tied to program participants’ entry-level skills. Those who needed little basic 
education to earn a GED were much more likely to be successful in this regard. This finding is 
also consistent with prior research, involving both mandatory and voluntary programs for school 
dropouts. 10 

Interestingly, motivation to participate in adult education programs did not affect program 
success in terms of GED receipt. Even though some sample members indicated that they did not 
want to attend school, those who made this claim as they entered the programs and were required 
to participate anyway experienced substantial increases in GED receipt, just like sample mem- 
bers who did express a preference for adult education at program entry. This shows not only that 
welfare-to-work programs can induce individuals to do things they might not otherwise do but 
also that participants in adult education can be successful even if they prefer not to participate. 

B. Individual Efforts and School Effects: The Payoff to Participation in Adult 
Education (Chapter 3) 

Nonexperimental analyses in the three HCD programs discussed above (Atlanta, Grand 
Rapids, and Riverside) suggest that the amount of time that participants spent in adult education 
classes affected their educational attainment and achievement outcomes. However, these rela- 
tionships were not straightforward. During participants’ first year of participation in basic educa- 
tion, additional months of participation were not associated with higher literacy test scores. How- 
ever, after a year of participation, additional months in adult education appeared to substantially 
increase test scores (an increase of .55 of a standard deviation for six additional months of par- 
ticipation)." In this sample, this effect was comparable to the differences in baseline literacy 
scores associated with having attended an additional month of high school. These findings sug- 
gest that a threshold level of participation of approximately one year is needed to achieve mean- 
ingful literacy gains lasting until the test administered in the two-year follow-up survey. (There 
could be a significant lag between the end of participation and this survey, which could make it 
difficult to reliably identify more modest gains in basic skills.) 

For measured math skills, this relationship looked markedly different. Increases in math 
skills were associated with additional months of basic education during the first six months only. 
After that, no further increases in these skills were found. This suggests a “plateau” rather than a 
“threshold” type of relationship. Such a pattern could reflect limitations in the math skills being 
taught in adult education classes. 

The relationship between time in adult education and GED receipt followed a similar pat- 
tern. Additional months of participation increased the likelihood of GED receipt during the first 
six months of participation but not thereafter. 



l0 See, for example, Quint et al., 1997, and Martinson and Friedlander, 1994. 

n A standard deviation is a statistic capturing the variability of a particular measure or outcome in a sample. Ef- 
fects on test statistics and other education outcomes are often expressed as a percentage of a standard deviation, be- 
cause absolute changes in these outcomes are difficult to interpret. 
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When these analyses were conducted separately for program group members and control 
group members, it appeared that both groups experienced similar patterns of gains in literacy and 
math skills. This is interesting, because participation for program group members was manda- 
tory, while control group members sought out adult education services on their own initiative and 
participated in them voluntarily. One might expect the payoff to the control group to be greater, 
but no such difference was found. A difference was found for GED receipt, but it was the oppo- 
site: program group members were more likely than controls to receive a GED as a result of addi- 
tional participation in adult education. 

When these analyses were conducted for different groups of sample members defined us- 
ing baseline characteristics, little variation in the estimated effects of additional education on lit- 
eracy and math skills was found. Sample members who faced greater barriers to participation or 
who were less motivated to participate in adult education benefited from additional education in 
similar ways as sample members who did not face these barriers. However, the analyses also 
found that the relationship between adult education participation and GED receipt varied across 
the subgroups. The increased probability of earning a GED from short-term participation, as 
mentioned above, did not hold up for sample members entering the program having completed 
less than ninth grade. Conversely, it was found that, for those with low initial skill levels, the 
likelihood of GED receipt continued to increase with additional participation beyond six months. 
This is unsurprising, because one might expect longer-term participation in adult education to 
help those who started out with low skills more. 

Last, we analyzed how the relationship between time spent in adult education and educa- 
tion outcomes was affected by differences in provider characteristics, including measures such as 
class size, teachers’ experience and education levels, individual attention, the strength of the link 
between the education provider and the welfare-to-work program, and program exit standards. 
The reliability of this analysis was limited, because we were unable to study the links between 
individual teachers and individual students, instead having to correlate student outcomes with 
school-level averages of teacher characteristics. However, within these limitations, teachers ex- 
perience and teachers’ education each appeared to enhance significantly the payoff to additional 
time spent in education classes in terms of reading and math skills. No provider characteristics 
were identified that affected programs’ effectiveness in increasing GED receipt. (However, pro- 
vider data were limited, and the same caveats about our ability to match students to specific 
teachers applies here.) 

C. Does the Low- Wage Labor Market Value Basic Education? Effects of GED 
Receipt and Literacy Gains on the Self-Sufficiency of Welfare Recipients 
(Chapter 4) 

Having analyzed how participation in adult education programs affects literacy, math 
scores, and GED receipt, the next step (in Chapter 4) is to analyze the extent to which improve- 
ments in these education outcomes affect welfare recipients’ employment outcomes and welfare 
receipt. If those effects are strong, improving welfare-to-work programs’ ability to improve edu- 
cation outcomes would also improve the programs’ effectiveness in terms of employment and 
welfare receipt. However, this would not be the case if these education outcomes had little effect 
on sample members’ employment and welfare receipt. In that case, a focus on immediate em- 
ployment might be more appropriate. 
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The analyses presented in Chapter 4 also contribute to our knowledge about the value of 
the GED and the importance of basic skills in the low-wage labor market. Much of the GED- 
related research has focused on comparisons of GED holders and high school graduates, using 
national data sets. Those findings may have limited relevance for programmatic choices made on 
behalf of welfare recipients, whose needs and experiences may be different from those of other 
school dropouts. 

GED receipt appeared to substantially increase earnings. The estimated effect on annual 
earnings in the third year of follow-up was approximately $771. GED recipients also received 
fewer welfare benefits (an estimated reduction of $331). These estimated effects remained largely 
unchanged when measures of time spent in adult education or of reading test scores were intro- 
duced as control variables into the analysis. This suggests that our estimates of the value of this 
credential reflect the effects of the credential itself, not the underlying basic skills or participation 
in adult education programs. It also was found that earning a GED had stronger estimated effects 
for program group members than for control group members. This suggests that the other aspects 
of the welfare-to-work programs (ranging from the program’s message to services like job search 
assistance, skills training, and college programs) further enhanced the value of this credential by 
increasing participants’ ability to make use of their newly acquired credential in the workplace. 

Like all analyses of educational attainment, analyses of GED receipt are potentially af- 
fected by selection bias. Such bias occurs when recipients of GED credentials are different from 
nonrecipients in ways that are not controlled in the analysis. In Chapter 4, effects of GED receipt 
were estimated in various ways to assess the sensitivity of the findings to selection bias and other 
problems. In general, the different estimates were consistent with one another, and there was no 
evidence that uncontrolled differences in motivation or ability explained the apparent effects of 
GED receipt on earnings and self-sufficiency. However, the findings presented in Chapter 4 
could not be confirmed with an advanced statistical method (an “instrumental variables” estima- 
tor) because statistical precision was lacking. 

Analyses of the effects of greater reading skills on employment outcomes and self- 
sufficiency found those effects to be substantial. An increase of one standard deviation in reading 
scores was associated with $355 in additional earnings during the year following the test (the 
third year of follow-up). This effect was independent of (that is, in addition to) any effect from 
earning a GED credential. (Introducing math skills separately did not show an independent effect 
separate from that associated with greater reading skills.) All this suggests that HCD programs 
could have more substantial effects on the economic outcomes of welfare recipients if these pro- 
grams managed to improve their effects on mediating education outcomes. Our analyses suggest 
that increased retention might be one way to achieve this. Research on “best practices” in adult 
education for welfare recipients has suggested that programs’ ability to retain students and im- 
prove their skills is affected by many program characteristics, including: 12 

® developing a well-defined mission, 

° providing specially targeted classes to students who are welfare recipients, 



l2 Adapted from Quint, 1997, p. 10. 
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o having skilled, experienced teachers, 
o emphasizing staff development, 

o adopting varied instructional approaches, including small group and computer 
activities, 

o communicating frequently with welfare-to-work program staff, 
o stressing regular attendance, 
o aggressively following-up on absences, 
o adopting relatively intensive class schedules, and 

o promoting a high degree of teacher-student and student-student interaction. 

Finally, a greater emphasis on identifying and addressing learning disabilities, which now 
remain largely undiagnosed, could greatly improve programs’ ability to serve their students suc- 
cessfully. 

D. Beyorad Basic Education: The Benefits of Skiils Training and College 
(Chapter 6) 

The last chapter of this report describes the effects of postsecondary education and train- 
ing, focusing on participants in adult education programs across all 1 1 programs included in the 
NEWWS study. The chapter analyzes who among these participants go on to postsecondary edu- 
cation and training and how such postsecondary participation affects their employment outcomes 
and self-sufficiency. 

The chapter reports that relatively few of those who participated in adult education pro- 
grams (15 percent of participants) made it to postsecondary services during the two-year follow- 
up period. Among adult education participants, those who earned a GED and did so in a rela- 
tively short amount of time (less than a year) were most likely to enter postsecondary programs. 
These sample members also were more likely to have entered the NEWWS study with higher 
initial literacy skills and having completed more grades in high school. 

Once enrolled in skills training or college programs, postsecondary participants remained 
in these programs for about seven months on average. More than 40 percent participated longer 
than six months, and almost 15 percent were enrolled longer than a year. At the time of follow- 
up, 29 percent of postsecondary participants were still enrolled in these programs, which means 
that these participation figures are preliminary and that participation could turn out to be longer 
on average when additional follow-up survey data become available. For those entering educa- 
tion programs without a high school credential, participation in postsecondary programs often 
follows a spell of adult education, suggesting that successful HCD programs for welfare recipi- 
ents without high school credentials may require more time than many states’ welfare time limits 
allow. 



In nonexperimental analyses, participation in postsecondary programs was found to have 
substantial benefits in terms of greater earnings and lower welfare receipt. These benefits did not 
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appear until after sample members completed their education and training. Effects for postsec- 
ondary participants appeared in the third year following their initial adult education spell. In that 
year, their earnings were $1,542 (or 47 percent) higher than those of sample members who re- 
ceived only adult education, while their welfare benefits were $919 (or 32 percent) lower. These 
estimated effects were not contingent on participants completing their spell of postsecondary 
education or training with a credential or certificate. 



III. Conclusions and Implications for Policy 

A. The Challenge of Making Adult Education Work for Welfare Recipients 

This study of adult education for welfare recipients who do not have a high school cre- 
dential uncovered several different patterns of effects. Assignment to a Human Capital Develop- 
ment program had substantial impacts on these welfare recipients’ participation in adult educa- 
tion, modest impacts on their GED receipt, and no impacts on measured literacy and math skills. 
The study also found that, within a three-year follow-up period, the effects of HCD programs on 
earnings and welfare receipt were positive but limited, especially compared with labor force at- 
tachment programs. 

A second pattern of findings concerns the dynamics of participation, learning, graduation, 
and skills acquisition that underlie the experimental impact story. The exploration of this pattern 
begins (in Chapter 3) with a discussion of one of the key questions underlying education and 
learning: What is the value of additional instruction? In addressing this question for three educa- 
tion-focused programs in Atlanta, Grand Rapids, and Riverside, Chapter 3 uncovers that addi- 
tional months in school (our only reliable measure of additional instruction) matter; they increase 
literacy, math skills, and GED receipt. However, long-term participation (longer than a year) is 
necessary to achieve a measurable payoff in increased literacy skills, while increases in math 
skills and GED receipt seem to be limited to the first six months of instruction. 

At the same time, the average program group member in these three programs received 
only about 244 hours (or about twelve 20-hour weeks) of adult education, which is substantially 
less than one year of high school — insufficient to make up the education deficit with which most 
of these welfare recipients entered the programs. This could explain why only 1 1 percent of pro- 
gram group members earned a GED during the follow-up period and why fewer than 1 5 percent 
of participants in adult education went on to skills training and college programs. 

Next, using nonexperimental methods, the report shows that for those who did reach 
these milestones, participating in adult education was beneficial. Chapters 4 and 6 show that 
payoffs from GED receipt, increased literacy skills, and postsecondary education and training 
were substantial. Thus, in summary, the analyses presented in this report confirm the internal 
logic underlying the human capital program model (as outlined earlier in Figure 1.1). However, 
too few program group members made it through the different steps to experience the anticipated 
payoffs at the end. Assuming that other program participants would experience comparable bene- 
fits, the challenge facing welfare-to-work program administrators and adult education providers 
is to find a way to retain more students long enough, to help them reach the immediate goal of 
earning a GED, and to help them access postsecondary services that allow them to capitalize on 
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this credential. This could occur while people are receiving welfare benefits or after they have 
left the welfare rolls. However, there is no guarantee that simply increasing the duration of par- 
ticipation is sufficient to reach educational goals. The quality of instruction, the appropriateness 
of the material and technology for participants with low skills and possibly with learning disabili- 
ties, and the sometimes limited motivation of program participants are likely important as well, 
although our study did not examine these factors. 

B. Policy Implications 

o Education-focused interventions for welfare recipients with low basic 
skills can improve these skills and increase GEB receipt. However, im- 
provements in these outcomes may require long spells off participation in 
adult education programs. 

The analyses presented in this report show that the long-term payoffs of an education- 
focused approach for welfare recipients without a high school diploma or GED can be substan- 
tial. However, the report also shows that it is a challenge to improve the basic skills and educa- 
tional attainment of welfare recipients, even in programs that are directly focused on education 
outcomes. For those entering with low skills and lacking years of high school, several years of 
basic education and postsecondary education may be needed to promote long-term success and 
self-sufficiency. In the current welfare environment, such a long-term commitment carries some 
risk inasmuch as long-term participation may exhaust welfare recipients’ limited allotted time on 
welfare. \ 

© For welfare recipients who are within easy reach of earning a GEB, pur- 
suit of such a credential is a good program option that produces substan- 
tial benefits, increasing welfare recipients’ earnings and their access to 
postsecondary education or training. 

Success is easier to achieve for those who are close to passing the GED test. Our findings 
suggest that this credential is a worthwhile short-term program goal, especially if it is combined 
with a targeted postsecondary activity. Together, a GED and skills training greatly increased 
earnings and reduced welfare receipt in the third year of follow-up. Education-focused welfare- 
to-work programs may be most successful when they can combine GED preparation and postsec- 
ondary services in a relatively short and intensive program (an option that, however, will not 
work with the most educationally disadvantaged). 

® Too few adult education students and GED recipients continue on to 
postsecondary education or training. Links between adult education pro- 
grams and postsecondary programs could be strengthened, and adult 
education students should be made aware of the limitations of having just 
a GEB credential as a way to improve one’s employment outcomes. 

Although GED receipt and increased basic skills appear to have positive effects on the 
earnings of welfare recipients, those effects appear to be much stronger when spells of adult edu- 
cation and receipt of a GED are followed by enrollment in postsecondary education or training 
programs. The orientation of many of these programs toward specific jobs and career opportuni- 
ties may be a factor in explaining these programs’ apparent benefits. Participation in postsecond- 
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ary education or training carries a price in terms of lower short-term earnings, but the longer-term 
effects of these services are substantial. Especially after a long spell of participation in adult edu- 
cation, it makes sense to cap off this investment with some college or vocational skills training. 

0 The analyses presented in this report address important questions about 
adult education provided to welfare recipients, but many important ques- 
tions remain, especially regarding the quality of participation in adult 
education, the appropriateness off adult education services provided to 
welfare recipients, and the possible benefits off education and training for 
those who do have an education credential when they enter the programs. 

In this report, we present program effects on participation in adult education and key edu- 
cational outcomes. We relate these effects to one another and to the employment and welfare 
outcomes of those being targeted by the programs. However, although we assessed the effects of 
participation in adult education, we did not capture all the reasons for wowparticipation. Although 
we measured the effects of earning a GED, we do not know why so many participants never re- 
ceived this credential. Answers to both of these questions, and others like it, may provide a 
greater understanding of the “quality” of participation (that is, the actual commitment to learning 
manifested by students who were coerced to participate in adult education) and by the quality of 
the instruction (that is, the appropriateness of teaching materials and techniques for the welfare 
recipients in these programs). The study data did not capture either one of these “quality” meas- 
ures accurately, and it is therefore difficult to say whether simply increasing enrollment in adult 
education programs beyond current levels would significantly improve the outcomes of these 
programs. More detailed data about the quality of students’ program experience must be col- 
lected to address these questions, and more systematic comparisons of the different types of adult 
education programs that serve welfare recipients are needed. 

The analyses presented in this report are limited to welfare recipients who did not have a 
high school diploma or GED and were considered to be in need of basic education. No parallel 
study was conducted to examine the effects of education and training on welfare recipients who 
did have an education credential. Our findings do not generalize to this group, and additional re- 
search may be needed to assess whether and how additional education and training benefits wel- 
fare recipients who are less disadvantaged academically. 
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Chapter 2 

Characteristics off Admit Education im Welffare-to-Work Programs 

and 

tine Effects offEdmcatiom-Focmsed Welffare-to=Work Programs 
om Edmcatiomal Attaimmemt and Achievement 



I. EmtrodiLictioini 



The purposes of this chapter are to describe the adult education that was provided as part 
of three education-focused welfare-to-work programs that were run in the early to mid 1990s and 
to present estimates of the effects of the programs on educational outcomes. 1 The chapter uses 
data from three education-focused mandatory welfare-to-work programs studied as part of the 
National Evaluation of Welfare-to-Work Strategies (NEWWS Evaluation). 2 As mentioned in 
Chapter 1, three of the sites — Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, Califor- 
nia — operated two programs side by side: an employment-focused, or Labor Force Attachment 
(LFA) program; and an education-focused, or Human Capital Development (HCD) program. The 
education-focused programs emphasized skills-building through education and training as a pre- 
cursor to employment, based on the belief that an upfront investment in the skills levels of wel- 
fare recipients allows them to obtain higher-paying and more secure jobs. In the education- 
focused programs, most people were first assigned to education or training; adult education was 
the most common activity because of the generally low educational attainment of the enrollees at 
program entry. 3 

This chapter answers the following questions: 

o What was the adult education like that was provided as part of the mandatory 
education-focused welfare-to-work programs in Atlanta, Grand Rapids, and 
Riverside? 

o What were the welfare-to-work programs’ effects on the use of adult educa- 
tion services? 

o What were the programs’ effects on educational attainment and achievement? 

o How did these effects vary for different types of welfare recipients (that is, 
subgroups of sample members)? 



'Recall from Chapter 1 that adult education encompasses adult basic education (ABE) classes, General Educa- 
tional Development (GED) classes, high school completion classes, and English as a Second Language (ESL) 
classes. 

2 The NEWWS Evaluation is a study of 1 1 welfare-to-work programs run in seven sites. The evaluation is being 
conducted by the Manpower Demonstration Research Corporation, under contract to the U.S. Department of Health 
and Human Services, with support from the U.S. Department of Education. 

3 See Hamilton et al., 1997, for a detailed comparison of the two program approaches in these three sites. 



The answer to the first question is essentially a description of the adult education pro- 
vided in the three programs, covering such issues as the links between the education institutions 
and the welfare-to-work programs, the classes’ educational content and methods, and the teach- 
ers’ qualifications. The next three questions shift the focus to the broader context of the welfare- 
to-work programs; the answers to the questions are based on analyses of various program effects. 
As discussed in Chapter 1, the NEWWS Evaluation used a random assignment research design. 
Outcomes for the control group represent what would occur in the absence of a mandatory wel- 
fare-to-work intervention. The differences between outcomes for the program group and the con- 
trol group represent estimated effects, or impacts, of the programs. 

Conclusions from a random assignment experiment are reliable and can provide impor- 
tant information on welfare recipients’ participation in adult education and the effects of manda- 
tory welfare-to-work programs on recipients’ educational outcomes. It should be noted, however, 
that the analysis in this chapter does not attempt to isolate the effects of the adult education itself; 
rather, it focuses on the effects of the entire package of the welfare-to-work programs’ services 
and mandates described below. Impacts are averaged over the full sample — those who partici- 
pated in adult education as well as those who did not. This can “dilute” the effects of adult edu- 
cation. (Chapter 3 explores more directly the connections between adult education and various 
outcomes.) 

The fact that the adult education examined here was provided in the context of mandatory 
welfare-to-work programs raises two important considerations. First, adult education was only 
part of the program treatment; the programs also included other activities, such as job search and 
vocational training, and they offered support services, such as child care assistance and counsel- 
ing. In addition, the programs were “mandatory” — they could impose a financial penalty on those 
who did not comply with program requirements. The mandates and financial sanctions, however, 
remained in effect only as long as a person was on welfare; someone who left the rolls could ter- 
minate her participation in all program activities, including adult education, without any penalty 
from the welfare-to-work program. Second, the programs involved adults in education who 
would not otherwise have participated. Whereas “traditional” students in adult education pro- 
grams enroll voluntarily and can therefore be presumed to be motivated to learn, such motivation 
cannot be assumed in the case of the students studied here. 

Typically, adult education programs have targeted adults without a high school diploma or 
General Educational Development (GED) certificate. 4 Therefore, this chapter focuses on this por- 
tion of the sample in each site (adults with a high school diploma or GED were also randomly as- 
signed to these programs, but they are not considered here). In other words, the full sample for this 
chapter is those who did not have a high school diploma or GED at random assignment. 

II. Summary off the Findings 

o Generally, the inclusion of welfare-to-work program students did not 
greatly affect the operations, curricula, or teaching methods of the adult 
education programs. 



4 Pauly, 1995. 
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Although the education programs made some changes to accommodate welfare-to-work 
program regulations and students, the adult education provided as part of the three welfare-to- 
work programs was similar to that provided to students who were not part of a welfare-to-work 
program. Most education institutions included some work preparation activities in classes, but 
they emphasized traditional adult education content (academic topics and preparation for the 
GED tests). The welfare-to-work programs, however, provided their students with additional 
support, such as child care. 

o Admit education in the three welfare-to-work programs was provided by 
a wide variety of education institutions. Although many differences exist 
when comparing the institutions, the differences do not appear to be sys- 
tematic. 

In other words, no distinct, separate categories of providers emerge based on the charac- 
teristics examined in this chapter. Likewise, differences exist across the three sites, but they are 
not systematic. 

o Like most mandatory education-focused welfare-to-work programs that 
have been studied, the programs in Atlanta, Grand Rapids, and Riverside 
substantially increased participation in adult education. 

Without the intervention of a mandatory welfare-to-work program, about one-fifth (18 
percent) of the control group took part in adult education classes over a two-year follow-up pe- 
riod. The programs studied here more than doubled this participation rate: one-half (50 percent) 
of the program group participated in adult education. The programs more than tripled the time 
spent in adult education: control group members spent an average of 68 hours in classes, com- 
pared with 244 hours for program group members — an increase of 176 hours. This increase in 
hours participated should be considered in the context of education outside the adult education 
system. As mentioned earlier, this increase is substantially less than the instruction provided in 
one year of high school and therefore should not be expected to produce dramatic changes in 
educational outcomes. Presented in terms of months, this means that control group members par- 
ticipated in adult education for an average of 1 .2 months, while program group members partici- 
pated for an average of 3.7 months — an increase of 2.5 months. 

o Mirrorlug the pattern found in other studies, the three welfare-to-work 
programs discussed in this chapter increased educational attainment but 
had no effect on educational achievement. 

Across the three sites, 4 percent of the control group received a high school diploma or 
GED during the two years following random assignment, compared with 1 1 percent of the pro- 
gram group — a modest increase of 7 percentage points. The programs did not, however, increase 
scores on standardized reading and math tests. Various factors may make it difficult for programs 
to produce test score gains. First, many who were in the programs did not receive any adult edu- 
cation; it is possible that education participants did achieve test score gains but that their gains 
are masked by outcomes for nonparticipants. Second, not all adult education is designed to in- 
crease reading and math achievement. Third, standardized tests may not measure the skills that 
were learned. Fourth, administration of achievement tests as part of a survey, rather than in a 
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classroom setting, may somewhat limit the reliability of test results, making it more difficult to 
find statistically significant program effects. 

o The findings presented in this chapter show that mandatory welfare-to- 
work programs can substantially increase participation in adult educa- 
tion and educational attainment for many different types of people. 

The programs increased participation for all 20 subgroups examined, and they increased 
high school diploma or GED attainment for 17 of the 20 subgroups. 

o There is no systematic variation in participation impacts across the sub- 
groups. 

A previous study (an evaluation of Cali fornia’s GAIN program) found that the welfare-to- 
work program generally had a greater effect on the amount of education received by the more 
disadvantaged segment of the caseload. Researchers proposed that this occurred in part because 
more disadvantaged people tended to spend more time on AFDC and were available to partici- 
pate in activities for a longer period. 5 In the present study, this pattern was found for only some 
of the subgroups. 

o No clear relationship exists between the impact on hours of participation 
in adult education and impacts on educational attainment or educational 
achievement. 

As was found in the GAIN evaluation, subgroups with larger increases in hours of par- 
ticipation did not necessarily have larger gains in attainment or achievement. Regarding educa- 
tional attainment, this probably reflects that many persons who received a high school diploma or 
GED were close to receiving one when they entered the programs, which then provided the nec- 
essary support services, the moral support, or the “push” needed to obtain the credential. Thus, 
their stays in education likely were relatively brief. Those who remained in education for ex- 
tended periods were probably far from receiving the credential. (Using more sophisticated statis- 
tical methods, Chapter 3 explores further the relationship between length of participation and 
educational attainment and achievement.) 

o As has been found In previous research, the programs studied here pro- 
duced much larger impacts on educational attainment for individuals 
with higher preprogram achievement levels than for those with lower 
preprogram achievement levels. 

The programs increased the receipt of a high school diploma or GED by 16 percentage 
points among those entering with high reading scores, and by 1 8 percentage points among those 
entering with high math scores (compared with impacts of 3 percentage points among those with 
low reading and low math scores). Said another way, the welfare-to-work programs helped many 
individuals with higher skills levels get education credentials. 



5 Martinson and Friedlander, 1994. 



® Notably, the programs produced major impacts for individuals who were 
not motivated to attend school. 

The programs substantially increased participation and receipt of education credentials for 
those who at program entry said they did not like school and/or did not plan to go to school. 

® The programs produced smaller impacts for individuals with many fam- 
ily or personal problems than for those in most other subgroups. 

The programs produced somewhat smaller participation increases for those who had fam- 
ily or personal problems (such as health or emotional problems) and did not increase educational 
attainment for this subgroup. 

III. A Brief Review of Frior Research 

Before discussing results from the three programs studied as part of the NEWWS Evalua- 
tion, it is useful to briefly review relevant prior research. This section summarizes findings on the 
adult education that has been provided to welfare recipients as part of education-focused welfare- 
to-work programs, participation in adult education in these programs, and various effects of these 
welfare-to-work programs. 6 

A. A Description of Adult Education for Welfare Recipients 

For decades adult educators have placed a high priority on serving people on welfare. Un- 
til the late 1980s, however, few direct linkages existed between adult education and welfare. In 
1988, the Family Support Act (FSA) for the first time closely linked adult education with welfare 
and provided substantial benefits (including child care and transportation support)as well as obli- 
gations for adult education students. Welfare-to-work programs that operated under the FSA re- 
ferred many people who were receiving cash assistance under the Aid to Families with Depend- 
ent Children (AFDC) program 7 to adult education, but many of these programs did not contribute 
to the funding of the education programs; this may have limited the nature and extent of coordi- 
nation between the adult education and welfare-to-work systems. Although coordination on refer- 
rals and attendance monitoring was widespread, coordinated efforts to improve program quality 
and effectiveness appear to have been less common. 

Research has identified considerable diversity in the goals of adult educators who served 
people on AFDC: some educators preferred approaches that tie adult education to work, while 
others focused on the traditional goals of higher reading and math achievement, English language 
fluency, and the receipt of the GED credential. Sometimes the goals of adult educators differed 
from the goals of welfare-to-work program operators or from the goals of adult students. One 
analyst 8 pointed out that failing to address such disagreements can lead to a failure to develop 



6 This section summarizes key points from a previous NEWWS Evaluation publication, The JOBS Evaluation: 
Adult Education for People on AFDC: A Synthesis of Research (Pauly, 1995). 

7 Until the 1996 welfare reform law, AFDC was the nation’s principal safety net for poor families. 

8 Pauly, 1995. 



educational approaches that meet the various goals and can result in services that do not meet 
students’ needs. 

Reflecting diverse goals, the content of adult education programs serving people on 
AFDC varied substantially. Many programs based their educational content primarily on pub- 
lisher’s textbooks, workbooks, and other materials. While many programs went beyond a “pub- 
lisher-driven, ‘plain vanilla’ approach,” 9 the prevalence of innovation in educational content is 
unknown. Studies have identified various kinds of work-related content in adult education pro- 
grams serving AFDC recipients — such as punctuality, styles of dress for the workplace, career 
planning activities, and job search instruction — although it is unclear how many programs in- 
cluded this type of content. Many adult education programs incorporated motivational and sup- 
portive content, such as self-esteem-building activities and study skills instruction. 

Research has not identified how widespread particular methods of teaching and learning 
are in adult education programs that serve welfare clients, and the available knowledge base is 
too limited to evaluate the effectiveness of specific instructional approaches and innovations. 
Based on the view that students learn in different ways, some programs have used multiple 
methods, often combining both group and individual activities, computer-assisted instruction, 
tutoring and other one-on-one instruction, and cooperative learning. 

The schedules of adult education programs serving people on AFDC also varied substan- 
tially: some programs offered 25 or more hours of instruction each week, while others offered 
fewer than 10 hours per week. These differences reflect differences in programs’ resource levels 
and intensity of services. 

Given the diversity of goals for welfare clients who participate in adult education, it is not 
surprising that there is little published information on the quality of adult education services for 
AFDC recipients. 

An earlier NEWWS Evaluation report profiled adult education programs in four commu- 
nities that were considered to be serving welfare clients “in innovative and promising ways.” 10 
The four programs, which were not in the sites formally involved in the NEWWS Evaluation, 
were chosen by MDRC researchers and U.S. Department of Education staff based on perceptions 
of innovation and quality in the education community, interviews, and in-person visits. Case 
studies of these programs uncovered a number of “promising practices” across the programs: 1 1 

o a well-defined mission 

o separate classes specifically for students from welfare-to-work programs 

o skilled, experienced teachers 

o an emphasis on staff development 



9 Pauly, 1995, p. 20. 

l0 Quint, 1997, p. vi. 

"See Quint, 1997, for a discussion of how the programs were chosen, descriptions of the programs, and more 
detail on these “promising practices.” 
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o varied instructional approaches that involve active learning 

o frequent communication about students’ progress between educators and 
welfare-to-work program staff 

o an emphasis on regular attendance, with aggressive follow-up of absences 

o relatively intensive class schedules (at least 20 hours per week) 

o a high degree of teacher-student and student- student interaction 

It is important to note that this list was generated from operational wisdom rather than from sta- 
tistical evidence that these practices consistently affect positive outcomes. (This chapter will ex- 
amine some of these dimensions among adult education providers in Atlanta, Grand Rapids, and 
Riverside. Chapter 3 directly explores the link between some of these facets and increased educa- 

1 9 

tional attainment and achievement.) 

B. Participation in Admit Education in Education-Focused Welfare-to-Work 
Programs 

Welfare-to-work programs have been an effective pathway to adult education; previous 
studies found that programs substantially increased levels of participation in adult education 
classes. Furthermore, students who participated in adult education classes as part of a welfare-to- 
work program received considerably more hours of service than other adult education students. 
Among adult education programs funded by the Adult Education Act of 1966, students received 
a median of 58 hours of service; 13 in adult education programs tied to welfare-to-work programs, 
participants typically received at least 100 to 200 hours of service. Two factors may help explain 
this finding: federal regulations in place before the 1996 welfare law created incentives for states 
to assign a substantial fraction of welfare-to-work program participants to education services for 
at least 20 hours per week, and the prospect of financial sanctions in mandatory programs pre- 
sumably induced some people to remain in education longer than they would have on their own. 

These positive effects on the length of time spent in adult education classes must be con- 
sidered in the context of education outside the adult education system: one year of high school 
entails more than 800 hours of instruction. 14 In general, 100 or 200 hours of adult education 
should not be expected to produce dramatic changes in educational attainment or achievement. 



l2 Note, however, that the evaluation of adult education providers in Atlanta, Grand Rapids, and Riverside was 
designed before the completion of the case studies. While some of the provider characteristics examined in the three 
sites are similar to some of the identified “promising practices,” they are not identical, and the evaluation was not 
designed to test these practices. 

'^Development Associates, 1994. 

,4 Researchers at the National Center for Education Statistics found that the average high school student received 
25.2 credits of instruction in 1998, of which 18 were academic credits (U.S. Department of Education, 2000). Based 
on this information, MDRC researchers calculated the total number of hours of academic instruction received by an 
average high school student to be approximately 3,240 hours. (The calculation assumes that 18 credits are equivalent 
to 18 one-year courses of approximately one hour per day and that an average school year has approximately 180 
days.) Dividing 3,240 hours by 4 years results in an estimate of 810 hours of academic instruction annually. 
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Despite the fact that welfare-to-work programs increased hours of adult education partici- 
pation beyond what normally would have occurred, researchers have identified a few factors that 
limit the amount of education that welfare clients received. First, poor attendance was a major 
problem in adult education programs that served AFDC recipients (although not necessarily a 
bigger problem than for adult education programs serving those who were not receiving 
AFDC). 15 Second, evidence suggests that many persons exited adult education before completing 
the program, 16 either because they became employed or left the welfare rolls for other reasons or 
because they became ill or faced other problems that caused them to drop out. 17 

C. Effects of EdMcatioim-Eociuised Welfare-to-Work Programs 

1. Receipt of a GED Certificate. Welfare-to-work programs emphasizing adult edu- 
cation have consistently and significantly increased the receipt of a GED certificate. Studies in- 
cluding analyses of preprogram achievement levels found that people with higher preprogram 
achievement levels had substantially larger impacts on GED attainment than those with lower 
achievement levels. Adults with lower achievement levels often took ABE or ESL classes that 
were focused on improving reading, language, and math skills, rather than GED classes that pre- 
pared them to take the GED test. Also, students in GED classes who had lower achievement lev- 
els were generally less successful than other participants in attaining the GED credential. 

Research has also found that these programs left a substantial proportion without a GED 
or high school diploma. A number of factors help explain this: (1) importantly, many persons in 
these programs did not participate in adult education and were not expected to obtain a GED cer- 
tificate; (2) many of the welfare-to-work programs also stressed noneducation activities and al- 
lowed participants to seek employment; (3) as mentioned, ABE and ESL programs generally are 
not intended to help students obtain these credentials; and (4) some program exits occurred when 
individuals left AFDC. Thus it may not be a reasonable goal that all welfare recipients who lack 
a high school diploma or GED certificate get a credential. 

2. Educational Achievement. Raising reading and math test scores appears to be a 
challenging goal for welfare-to-work programs providing adult education. A 1995 synthesis of 
research found that among studies that examined achievement, only two of nine programs raised 
test scores. 18 Analogous to the pattern in the GED attainment results, the impacts on achievement 
in one of these studies were concentrated among individuals with higher preprogram achieve- 
ment levels. 19 

It is important to note, however, that the tests typically used to assess the functional read- 
ing and math skills of adults are not aligned with the curricula used in adult education classes. 
Tests may or may not measure the skills that were learned by the adult education students who 



l5 Teachers providing adult education in Atlanta, Grand Rapids, and Riverside rated the attendance of welfare-to-work 
program students as similar to, or better than, the attendance of students not in the program (Quint and Walter, 1995). 

l6 About one-quarter of teachers in Atlanta, Grand Rapids, and Riverside estimated that half or more of their wel- 
fare-to-work program students stopped attending class before completing it (Quint and Walter, 1995). 
n Quint, 1995. 

18 Pauly, 1995. 

l9 This was the GAIN program in San Diego; see Martinson and Friedlander, 1994. 
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were studied. Additional caveats, similar to those about GED attainment, apply to the test score 
results. First, many persons in the studies’ samples did not receive any adult education. Even if 
adult education participants achieved substantial test score gains, their gains could be diluted by 
the outcomes of others who did not receive adult education. Second, not all adult education pro- 
grams are designed to increase achievement in reading and math. GED preparation classes typi- 
cally concentrate on helping students learn the discipline-based information and skills that are the 
subject of the GED tests. These classes often assume that students already possess adequate read- 
ing skills, and they generally are not intended or expected to affect reading achievement levels. 
Although one of the GED tests covers math, the test concentrates on algebra and geometry, rather 
than on the lower-level math skills that are measured in many achievement tests for adults. 

3. Employment and Welfare Receipt. The relationship between adult education in 
welfare-to-work programs and economic impacts is complex. Many welfare-to-work programs 
have increased employment and earnings and have reduced welfare receipt for clients generally 
regarded as needing adult education, but the role of adult education in producing these impacts is 
unclear. (Chapter 4 of this report explores the effects of receiving a GED and of increased educa- 
tional achievement levels on subsequent earnings patterns. Then Chapter 5 examines the overall 
impacts of the three education-focused programs — along with the other eight programs in the 
NEWWS Evaluation — on earnings and welfare payments.) 

IV. A Description of the Adult Education Provided as Part of the 

Education-Focused Programs in Atlanta, Grand Rapids, and Riverside 

This section primarily uses data collected from the major institutions providing adult edu- 
cation to welfare-to-work program participants in Atlanta, Grand Rapids, and Riverside. The data 
were gathered through surveys administered to teachers and through interviews with administra- 
tors at the same institutions. 20 (See Appendix A for a description of these and other data sources 
used in this chapter and for the associated sample sizes.) 

Adult education in the three welfare-to-work programs was provided by a wide variety of 
education institutions. (This section uses interchangeably the terms education institution and 
education provider.) Although these institutions differed in many ways, the differences do not 
appear to be systematic. In other words, no clear, distinct categories of providers emerge based 
on the characteristics examined in this chapter. Likewise, differences in provider characteristics 
emerged across the three sites but were not systematic. (Chapter 3 examines the role of some of 
the provider characteristics discussed here in affecting students’ educational attainment and 
achievement.) 

The adult education programs in Atlanta, Grand Rapids, and Riverside were solid, more 
or less typical adult education programs. Generally, the inclusion of students who were in a wel- 
fare-to-work program did not greatly affect operations, the curricula, or the teaching methods of 
the adult education programs. In Riverside, the link between the welfare-to-work program and 
the education programs was a bit stronger, largely because the education programs were directly 



20 Some of this information was presented in Hamilton et al., 1997. It is included here to provide a complete de- 
scription of the education institutions. 
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funded by the welfare-to-work program. Across the sites, almost no classes served only welfare- 
to-work students, and typically these students were in the minority. The programs provided some 
instruction on work preparation, but most focused primarily on traditional educational content. 
Classes spent more time on reading and writing activities than on math, and most programs pro- 
vided at least a moderate degree of individual attention and emphasized personal relationships. 
On average, classes were smaller in Atlanta and Grand Rapids than in Riverside. 

A. Major Institutional Education! Providers 

All sites relied principally on adult education programs operated through local school 
systems, although Atlanta also used community-based nonprofit organizations. The size of 
these institutions varied considerably in every site: some schools were small, with annual stu- 
dent enrollments of 120 or fewer, and some were large, with annual student enrollments of 
several thousand. 

B. Types of Adult Education Offered 

The three sites used each type of adult education to varying degrees. Because the sites relied 
mostly on existing educational resources within their communities, state and local education poli- 
cies primarily determined what kinds of classes were available. The state of Michigan funded high 
school completion but not GED programs; consequently, in Grand Rapids more students were en- 
rolled in high school completion than in Atlanta or Riverside. Most classrooms contained more 
than one type of student. For example, high school completion and GED students might attend the 
same class in Grand Rapids, and ABE and GED students might be in a single class in Riverside. 

C. Links Between Adult Education and the Welfare-to-Work Programs 

The Riverside welfare-to-work program was unique among the three sites in that it nego- 
tiated contracts with all the schools serving welfare-to-work clients, using welfare-to-work pro- 
gram funds to help pay for these adult education classes. Atlanta and Grand Rapids, in contrast, 
referred people to education providers funded by sources outside the welfare-to-work program 
(usually state and local education departments). The adult education programs in these two sites 
relied heavily on revenue from state education agencies (sometimes augmented by funds from the 
state welfare departments) to fund instruction to welfare-to-work participants. 

Riverside took advantage of its resources and its contracting authority to influence aspects 
of the education its program members received. The Riverside program established precise crite- 
ria for determining how clients would be placed in different education programs (ABE, GED, or 
ESL) and the duration of these assignments. The contracts between the welfare department and 
the education providers included incentive payments for providers that succeeded in getting cli- 
ents to make progress in and complete their education assignments. Compared with Riverside, 
the Atlanta and Grand Rapids welfare-to-work programs gave much more discretion to education 
providers. Once the welfare-to-work program staff determined that individuals needed adult edu- 
cation and referred them to a school, the education providers’ staff were responsible for placing 
clients in an appropriate ABE, GED, ESL, or high school completion class and for determining 
when students should exit. (See section E below, on exit standards, for details on how the wel- 
fare-to-work programs influenced exits from classes.) 
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A majority of adult education staff serving welfare-to-work program participants in At- 
lanta, Grand Rapids, and Riverside indicated that they increased the number of days or hours that 
they offered classes in order to accommodate welfare-to-work students. These increases were of- 
ten driven by the programs’ need to assign clients to an average of 20 hours of instruction and 
study time per week, as required by federal regulations. A few programs also enhanced the kinds 
of services they offered, adding, for instance, additional counseling for welfare-to-work students 
or more vocational or job-readiness instruction. Across the three sites, program administrators in 
Atlanta reported the greatest expansion in the services they offered, but teachers in Atlanta indi- 
cated that the changes were modest. 

Table 2.1 summarizes some characteristics of the education providers using data from the 
teacher survey. Provider-level measures are the average of the responses among the teachers as- 
sociated with a specific provider. Teachers’ survey responses corroborated that the link between 
the welfare-to-work program and the education institutions was stronger in Riverside than in the 
other sites. Teachers were asked a series of questions on communication with the welfare-to- 
work program, including how often they talked with program staff and the likelihood they would 
report student attendance problems to program staff. (See Appendix B for a list of the survey 
questions used to create this scale and the other scales presented in the table.) As Table 2.1 
shows, in Atlanta and Grand Rapids one-third or fewer of the education providers had a moderate 
or high level of communication with the welfare-to-work program. In Riverside, however, almost 
three-fourths of providers had a moderate or high level of communication with the program. Re- 
call that frequent communication about students’ progress between educators and program staff 
was identified as a “promising practice.” Based on teachers’ responses, this occurred more fre- 
quently in Riverside than in the other sites. 

Another “promising practice” mentioned earlier is providing separate classes for welfare- 
to-work students. No education institution in the three sites exclusively served welfare-to-work 
clients, and almost all classes included students from welfare-to-work programs as well as other 
adult education students. As Table 2.1 shows, fewer than 10 percent of providers in Grand Rap- 
ids and Riverside had a high proportion (70 percent or more) of welfare-to-work program stu- 
dents in their classes. Atlanta providers had classes with higher proportions of welfare-to-work 
students, but in only 29 percent of institutions did these students make up 70 percent or more of 
the typical class. 

D. Eduicatnomal Comteet arad Methods 

The education institutions, on average, spent more time on reading and writing activities 
than on math activities. As Table 2.1 shows, between 0 and 27 percent of institutions spent only a 
little time on reading and writing, compared with between 14 and 73 percent of institutions that 
spent a little time on math. According to teachers’ responses, instruction in Atlanta was most 
evenly balanced between reading and writing and math activities: 100 percent of providers spent 
a moderate amount or a lot of time on reading and writing, and 86 percent spent a moderate 
amount or a lot of time on math. 

Skills tested in the Test of Applied Literacy Skills (TALS) document literacy test were 
taught in classes in Atlanta and Riverside to a greater degree than in classes in Grand Rapids. 
Skills tested in the Comprehensive Adult Student Assessment System (CASAS) math test were 
taught in classes in all the sites. Despite the generally greater emphasis on reading instruction, 
providers did not emphasize reading test items more than they emphasized math test items. The 



National Evaluation of Welfare-to-Work Strategies 
Table 2.1 



Characteristics of Major Education Institutions Providing 
Adult Education to Sample Members in 
Three Education-Focused Programs 



Scale or Measure 


Atlanta (%) 


Grand Rapids (%) 


Riverside (%) 


Link to Welfare-to-Work Programs 








Communication between program and 
education institution staff 








High 


28.6 


0.0 


9.1 


Moderate 


0.0 


33.3 


63.6 


Low 


71.4 


66.7 


27.3 


Average proportion of welfare-to-work program 
students in the classroom 








70% or more 


28.6 


6.7 


9.1 


30% - 69% 


42.9 


33.3 


18.2 


Less than 30% 


28.6 


60.0 


72.7 


Educational Content and Methods 








Time spent on reading and writing activities 








A lot 


42.9 


13.3 


18.2 


A moderate amount 


57.1 


66.7 


54.6 


A little 


0.0 


20.0 


27.3 


Time spent on mathematics activities 








A lot 


28.6 


6.7 


0.0 


A moderate amount 


57.1 


33.3 


27.3 


A little 


14.3 


60.0 


72.7 


Degree to which skills tested in the TALS 
document literacy test were taught in class 








High 


71.4 


21.4 


70.0 


Moderate 


28.6 


21.4 


20.0 


Low 


0.0 


57.1 


10.0 


Degree to which skills tested in the CASAS 
math test were taught in class 








High 


71.4 


64.3 


100.0 


Moderate 


14.3 


21.4 


0.0 


Low 


14.3 


14.3 


0.0 


Education/ work link 








Strong 


0.0 


0.0 


9.1 


Moderate 


100.0 


60.0 


63.6 


Weak 


0.0 


40.0 


27.3 


Proportion of teachers who primarily used 








a mix of traditional and nontraditional teaching 
methods 8 








50% or more 


28.6 


53.3 


63.6 


Less than 50% 


71.4 


46.7 


36.4 
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Table 2.1 (continued) 



Scale or measure 


Atlanta (%) 


Grand Rapids (%) 


Riverside (%) 


Classroom Environment 
Degree of individual attention 


High 


14.3 


0.0 


0.0 


Moderate 


71.4 


86.7 


90.9 


Low 


14.3 


13.3 


9.1 


Emphasis on personal relationships 


High 


85.7 


86.7 


45.5 


Moderate 


14.3 


13.3 


54.6 


Low 


0.0 


0.0 


0.0 


Average number of students attending class 


1 8 or fewer 


100.0 


93.3 


18.2 


More than 18 


0.0 


6.7 


81.8 


Teachers' Characteristics and Benefits 
Average proportion of teachers working full time b 


50% or more 


71.4 


86.7 


36.4 


Less than 50% 


28.6 


13.3 


63.6 


Average years of teaching experience 


10 or more years 


57.1 


93.3 


100.0 


Less than 10 years 


42.9 


6.7 


0.0 


Percentage of teachers with master's degrees 
or higher education credentials 


50% or more 


57.1 


60.0 


63.6 


Less than 50% 


42.9 


40.0 


36.4 


Receipt of fringe benefits 0 


High 


14.3 


53.3 


9.1 


Moderate 


71.4 


46.7 


81.8 


Low 


14.3 


0.0 


9.1 


Teachers' Views 

Average rating of teaching materials and 
equipment 


High 


71.4 


53.3 


81.8 


Moderate 


28.6 


46.7 


18.2 


Low 


0.0 


0.0 


0.0 


Average rating of classroom morale 
High 


100.0 


86.7 


90.9 


Moderate 


0.0 


13.3 


9.1 


Low 


0.0 


0.0 


0.0 


Self-perceived teacher effectiveness 


High 


100.0 


86.7 


90.9 


Moderate 


0.0 


13.3 


9.1 


Low 


0.0 


0.0 


0.0 


Number of education institutions 


7 


15 


11 
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Table 2.1 (continued) 

SOURCE: MDRC calculations from data obtained through a survey of adult education teachers in each major 
education institution serving welfare-to-work clients in the fall of 1993. 

NOTES: An education institution's score on the above scales and measures represents the average of its 
teachers' scores. In Atlanta, 24 teachers were surveyed; in Grand Rapids, 79; and in Riverside, 45. For an 
enumeration of composite measures presented above, see Appendix B. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

a In this scale, traditional teaching methods are defined as published textbooks, workbooks, or worksheets, 
and nontraditional teaching methods are novels or short stories, newspapers or magazines, materials written by 
teachers or by other staff members at the institution, materials developed by students, materials that are about 
the students’ home and community environment, educational games, audio visual equipment, computers, and 
job application forms and other real-life documents. Teachers were classified as primarily using a mix of 
traditional and nontraditional methods if they indicated that the two materials they used most commonly in 
class included one traditional and one nontraditional method. 

b Full-time employment is defined as working 30 hours per week or more. 

c Fringe benefits are defined here as health benefits, vacation benefits, sick leave, and retirement benefits. 
In the scale above, "low" indicates that teachers at education providers, on average, received none of these 
benefits; "moderate" indicates that teachers received between one and three of these benefits; and "high" 
indicates that teachers received more than three of these benefits. 
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questions used to create the “test items” scales simply capture whether the skills covered by the 
test questions were taught in the classroom, not whether teaching these skills consumed most of 
the class time. 21 A class that spent a majority of time on reading could still have spent some time 
on skills covered in the math test. 

Most institutions attempted to integrate work preparation into their education curriculum, 
but almost no institutions did this to a high degree (that is, reported providing a strong link be- 
tween education and work). Activities used in classes to prepare students for work included 
composing letters to hypothetical employers, practicing writing resumes, and using reading mate- 
rial about work situations. 

As mentioned, an earlier study concluded that using a variety of teaching methods may be 
more effective than relying solely on traditional adult education methods (based on published 
textbooks, workbooks, or worksheets). 22 As Table 2.1 shows, more than half the providers in 
Grand Rapids and Riverside had teachers who used a mix of traditional and nontraditional teach- 
ing methods, compared with less than a third in Atlanta. For this measure, nontraditional teach- 
ing methods include novels or short stories, newspapers or magazines, materials developed by 
teachers or students, and computers (see the footnote for Table 2.1 for a complete list of nontra- 
ditional methods used). Teachers were classified as “primarily” using a mix of traditional and 
nontraditional methods if they indicated that their two most commonly used materials included 
one traditional and one nontraditional method. 

E. Exit Stamdlards 

As previously noted. Riverside established criteria for the duration of education assign- 
ments, and it offered the education providers incentive payments for getting clients to complete 
their assignments. Underlying these standards was a desire that program participants learn 
quickly, acquire just enough skills to move up to the next class level, and then leave the pro- 
gram’s education component. Riverside welfare-to-work staff, for example, were unlikely to al- 
low students starting out in an ABE or ESL class to remain in education until they earned a GED 
certificate. Rather, ABE or ESL students would have been permitted to stay in school only until 
they achieved a target score on an educational achievement test specified by the state welfare 
agency. After that they were expected to participate in program activities more directly related to 
finding entry-level employment. Riverside welfare-to-work staff generally expected participants 
in ABE, GED, and ESL to complete their education assignment within 6 to 12 months. 

In contrast, in Atlanta and Grand Rapids, both school staff and welfare-to-work staff usu- 
ally encouraged people beginning in ABE or ESL to stay in education until earning their high 
school diploma or GED certificate. Education providers in Atlanta and Grand Rapids sometimes 
recommended that individuals who had attained a high school diploma or GED certificate remain 
in an adult education classroom for a bit longer if they believed individuals were weak in an aca- 
demic subject area or needed to build more confidence before moving into college or vocational 



2l Teachers were asked to examine a series of questions that appear on the TALS document literacy test and the 
CASAS math test. They were then asked whether the skills and content area covered by the test questions were 
taught in their class. 

22 Quint, 1997. 





training. Exit criteria, therefore, were based much more on teachers’ assessments than on prede- 
termined standards or test scores. Adult education administrators estimated that it could take as 
long as three years for students to complete an education program in their schools, although wel- 
fare-to-work case managers in the two sites were asked to limit education classes to two years. 

F. Classroom Environment 

The four programs identified as “innovative and promising” in the earlier NEWWS 
Evaluation case studies were characterized by a high degree of teacher-student interaction. 
Most programs in Atlanta, Grand Rapids, and Riverside provided only a moderate degree of 
individual attention when instructing students (indicated by such practices as one-on-one in- 
struction and developing lesson plans for individual students); few provided either a high or 
low degree. (See Table 2.1.) Most providers did emphasize the importance of personal interac- 
tion between students, their teachers, and their fellow classmates; this was particularly true in 
Atlanta and Grand Rapids. 

Classes in Riverside, on average, were much larger than classes in the other two sites. 
Most classes in Riverside had more than 18 students, whereas all classes in Atlanta and most in 
Grand Rapids had 18 or fewer. 23 The differences in class size may partly reflect differences in 
how classes were organized at the three sites. Although a large majority of classes at all the sites 
operated on a fixed class schedule, Riverside had more classes than Atlanta or Grand Rapids that 
operated on a “drop-in” basis. The drop-in centers provided individual workstations that had 
computers and other learning material and could accommodate more students than could a formal 
classroom setting that operated on a fixed schedule. 

On average, classes met four days per week in Atlanta and Riverside and three days per 
week in Grand Rapids. The average number of class hours per week varied across the sites: 
classes in Atlanta met for an average of 16 hours per week; in Grand Rapids, 10 hours per week; 
and in Riverside, 1 7 hours per week (not shown in the table). The differences do not necessarily 
reflect different time spent overall in classroom activities. For example, Grand Rapids students 
often were assigned to more than one class — a product, possibly, of a curriculum that contained 
numerous high school courses. Adult education classroom time often was supplemented by sev- 
eral hours of independent study in all three sites. Few classes operated on a strict calendar; in- 
stead, students could enter and exit at any point during the school year. 24 

G. Teachers’ Characteristics, Benefits, and Views 

In Atlanta and Grand Rapids, at least half the teachers were employed full time at the vast 
majority of providers, whereas in Riverside only about a third (36 percent) of providers had more 
than half their teachers working full time (at least 30 hours per week). Having skilled, experi- 
enced teachers is one of the “promising practices” noted earlier. Most teachers had at least 10 



23 Average class attendance in Atlanta was 13 students; in Grand Rapids, 14 students; and in Riverside, 28 stu- 
dents (Hamilton et al., 1997). 

24 As noted earlier, a majority of the education institutions serving welfare-to-work program clients in Atlanta, 
Grand Rapids, and Riverside indicated that they increased the number of days or hours that they offered classes in 
order to accommodate these students. A study of adult education programs found that the mean weekly hours per 
week that clients attended class varied from 5 to 13 hours per week (Development Associates, 1992). 
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years of teaching experience, and in the majority of providers in each site at least half had a mas- 
ter’s degree or higher. Most providers offered teachers at least some fringe benefits — including 
health, vacation, and retirement benefits as well as sick leave. More providers in Grand Rapids 
than in the other sites offered a high level of benefits. 

Generally, teachers were satisfied with the materials and equipment available to them, in- 
cluding the physical plant, books, workbooks, computers, and software. MDRC field researchers 
usually shared this impression. Classroom morale was high in each of the sites, and the teachers 
perceived themselves to be effective. 



V. Characteristics off Welfare Recipients Without a High School Diploma 
or GED Certificate in the Three Sites 

Expanding the focus from the adult education providers to the welfare-to-work programs, 
the remainder of this chapter describes the characteristics of sample members in the study and 
discusses program effects on their participation in adult education, their receipt of educational 
credentials, and their educational achievement. Most analyses are presented for a single, pooled 

25 

research sample across the three sites. 

Initially, all those randomly assigned for the NEWWS Evaluation in Atlanta, Grand Rap- 
ids, and Riverside were mandated to participate in the welfare-to-work program. After random 
assignment, those assigned to the control group were excused from participating, while those in 
the program group faced possible reductions in their welfare grant if they failed to participate. 
Our sample is limited to single parents. 26 Among single parents, AFDC applicants and recipients 
were required to participate if their youngest child was 3 or older in Atlanta and Riverside, and 1 
or older in Grand Rapids, 27 and if they did not meet certain exemption criteria. 28 All sample 
members in Atlanta and Riverside were age 20 or older, while the Grand Rapids sample also in- 
cludes individuals of 18 and 19. 

As mentioned previously, this chapter examines the experiences of participants who 
are the typical target group for adult education: those without a high school diploma or GED 
certificate (at program entry). In Riverside, some sample members who had a high school 
diploma or GED certificate were included in the sample, namely, those who had low reading 



25 Programs were weighted equally in the pooled analyses. 

26 In Riverside, the primary wage earner in two-parent families was randomly assigned, but this portion of the 
sample is not analyzed in this chapter. 

27 Federal regulations mandated participation from parents with children 3 and older and gave states the option to 
mandate participation from parents with children as young as 1 . 

28 Federal exemption reasons included working 30 hours or more per week, having a disabling illness, being in at 
least the second trimester of pregnancy, or living in a remote area that made program activities inaccessible. Grand 
Rapids (Michigan) had some additional state-specific exemption reasons: if a recipient had three or more children 
under age 1 0, had been within the last five years a resident of a mental institution, had been using prescribed medica- 
tion for mental illness, or had been enrolled in a rehabilitation program for at least 1 5 hours per week. 
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or math skills or were not proficient in English. For ease of reference, all these individuals 
are referred to as those without a high school diploma or GED, or nongraduates. Among edu- 
cation-focused welfare-to-work programs included in the NEWWS Evaluation, this nongradu- 
ate sample comprised 39 percent of those in Atlanta, 40 percent of those in Grand Rapids, and 
100 percent of those in Riverside. 30 

Table 2.2 presents key demographic, educational, and employment information for the 
nongraduate samples pooled across the three education-focused programs. (The pooled numbers, 
weighting each site equally, represent the average for each measure across the three sites. Appen- 
dix Table C.l shows these characteristics broken down by site.) As the table shows, most sample 
members are women, and their average age at the beginning of the study was 31 years. About 
half (49 percent) of the sample is black, about one-third is white (31 percent), and roughly one- 
fifth (18 percent) is Hispanic. Sample members had an average of about two children. 

Eight percent of the sample had a high school diploma or GED certificate at program en- 
try (by design, all these individuals were in the Riverside sample), and the average grade com- 
pleted across the three sites was 10th grade. Twenty-three percent of the sample had recently 
been enrolled in education or training, and 17 percent were enrolled at program entry. 

• 1 1 

Achievement tests were administered at the point of random assignment. As the table 

shows, 63 percent of the sample scored at Level 1 or 2 on the reading test (Level 1 categorizes 
the lowest test scores; Level 5, the highest), and 66 percent scored at Level 1 or 2 on the math 
test. The National Adult Literacy Survey, conducted in 1992, found that in the nation as a whole, 
72 percent of people on AFDC scored at Levels 1 or 2 on a document literacy test whose scores 
are analogous to those obtained from the TALS test. Nationwide, 51 percent of all adults scored 
at Levels 1 or 2 on the national document literacy test. 32 

More than half (58 percent) of sample members in the three sites had ever worked full 
time for six months or more for one employer, and about one-third (3 1 percent) had earnings in 
the year before entering the program. Few sample members were employed when they entered 
the program. A majority had substantial prior AFDC receipt: 66 percent of the sample had re- 
ceived AFDC for two years or more. About one-third (31 percent) of the sample were living in 
public, subsidized, emergency, or temporary housing when they entered the study. 



29 Califomia limited participation in adult education: only those considered to be “in need” of education — those 
without a high school diploma or GED, those scoring below 2 1 5 on the GAIN Appraisal reading or math test, and 
those who were not proficient in English — could be assigned to education as a first activity. 

30 See Freedman et al., 1999, for the complete sample sizes. Note that California regulations prevented anyone 
who was not “in need” of education from being randomly assigned to the education-focused program. 

3 'in all three sites sample members took the CASAS math test. In Atlanta and Grand Rapids, sample members 
took the TALS document literacy test. In Riverside, sample members took the state-mandated GAIN Appraisal liter- 
acy test, developed by CASAS. Riverside’s literacy scores were converted to TALS scores using a “crosswalk” algo- 
rithm developed by researchers at Boston College (Haney et al., 1996). See Appendix D for information about the 
tests. 

32 Kirsch et al., 1993; cited in Pauly, 1995. 
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National Evaluation of Welfare-to-Work Strategies 
Table 2.2 

For Sample Members Without a High School Diploma or GED at Random Assignment: 
Selected Characteristics of Sample Members, Pooled Across 
Three Education-Focused Programs 



Characteristic 


Sample Members 


Demographic Characteristics 
Gender (%) 


Male 


5.9 


Female 


94.1 


Age (%) 


Less than 19 


J.D 


19-24 


21.0 


25-34 


46.6 


35-44 


22.9 


45 and over 


6.1 


Average age (years) 


30.8 


Ethnicity (%) 


Black 


4 V.l 


Hispanic 


17.7 


White 


30.5 


Other 


2.8 


Family Status 
Marital status (%) 


Never married 


53.0 


Married, living with spouse 


4.9 


Separated 


23.8 


Divorced 


16.8 


Widowed 


1.5 


Number of children (%) 


1 


37.0 


2 


32.8 


3 or more 


30.2 


Average number of children 


2.1 


Age of youngest child (%) 


2 or under 


18.8 


3 to 5 


36.3 


6 or over 


44.9 


Education Status 


Received high school diploma or GED certificate (%) 


7.5 


Average highest grade completed 


10.0 


Highest grade completed 


8 or below 


14.1 


9 or 10 


38.5 


1 1 or above 


47.4 



(continued) 
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Table 2.2 (continued) 



Characteristic 


Sample Members 


Enrolled in education or training in past 




12 months (%) 


23.3 


Currently enrolled in education or training (%) 


17.3 


Reading and Math Skills 




TALS document literacy test 3 




Scored at (%) 




Level 1 


19.3 


Level 2 


43.6 


Level 3 


28.2 


Level 4 


8 Q 


Level 5 


o.y 

0.0 


Level 1 - 2 


67 9 


Level 3 - 5 


37.1 


CASAS math test 




Scored at (%) 




Level 1 


18 7 


Level 2 


io. / 

47.2 


Level 3 


21 S 


Level 4 


m 1 1 J 
12.6 


Level 1 - 2 


fiS Q 


Level 3-4 


UJ.7 

34.1 


Labor Force Status 




Worked full time for 6 months or more 




for one employer (%) 


58.2 


Any earnings in past 12 months (%) 


30.6 


Currently employed (%) 


7.1 


Public Assistance Status 




Total prior AFDC receipt 6 (%) 




None 


n s 


Less than 1 year 


21.6 


1 year or more but less than 2 years 


1 1.7 


2 years or more but less than 5 years 


25.7 


5 years or more but less than 10 years 


18.9 


10 years or more 


21.6 


Raised as a child in a household receiving AFDC (%) 


32.2 


First spell of AFDC receipt 0 (%) 


17.4 


Housing Status 




Current housing status (%) 




Public housing 


16.0 


Subsidized housing 


13.3 


Emergency or temporary housing 


1.5 


None of the above 


69.3 


Sample size 


5,863 



(continued) 
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Table 2.2 (continued) 



SOURCES: MDRC calculations from information routinely collected by welfare staff and from TALS 
document literacy test and CASAS math test data. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited 
English, and thus were determined to be in need of basic education. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally in the pooled estimates. 

Distributions may not add to 100 percent because of rounding. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

TALS document literacy test scores for Riverside are based on scores earned on the Greater Avenues for 
Independence (GAIN) Appraisal literacy test and are converted to their TALS equivalent. 

b This refers to the total number of months accumulated from one or more spells on an individual’s own or 
spouse’s case. It does not include AFDC under a parent s name. 

‘This does not mean that such individuals were new to the AFDC rolls, only that this was their first spell 
on AFDC. This spell, however, may have lasted several years. 
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VI. Participation im Adult Education 

As discussed earlier, this chapter presents program effects from an evaluation that em- 
ployed a random assignment research design. In such a study, outcomes for control group mem- 
bers represent what would have occurred in the absence of the welfare-to-work intervention be- 
ing evaluated. In this section, then, participation in employment and training activities by control 
group members represents the level of participation that occurred without the intervention of the 
Atlanta, Grand Rapids, and Riverside programs. The difference between the participation level of 
the program group and the participation level of the control group indicates the impact of the 
programs — the additional participation that the programs induced. Participation estimates pre- 
sented in this section are based on 2,923 sample members’ responses to a survey administered 
about two years after random assignment. 

A. Participation in Program Activities for the Full Sample 

As previous NEWWS Evaluation publications have reported, the education-focused 
programs increased the level of participation in education, training, and employment activities 
(see Appendix Table C.2 for participation results broken down by site). 33 As Table 2.3 shows 
(pooling data across the three programs), 28 percent of the control group participated, for at 
least one day, in at least one employment or training activity within two years of entering the 
program. In other words, without the intervention of the welfare-to-work programs, 28 percent 
of welfare recipients took part in employment-related activities. The programs substantially 
increased this level of participation: 63 percent of the program group participated — an impact 
of 35 percentage points. 34 

Reflecting the programs’ focus on education, the largest program effects presented in 
Table 2.3 were on education and training activities. Owing to the low educational attainment 
and achievement of the nongraduate sample at program entry, the largest increases were in 
adult education: 50 percent of the program group participated in adult education — a 33 per- 
centage point impact above the control group participation rate of 1 8 percent. This impact re- 
flects a 30 percentage point increase in participation in ABE or GED classes (the survey did 
not distinguish between these two types of activities) and a 2 percentage point increase in par- 
ticipation in both ESL and high school completion classes. The 33 percentage point increase in 
adult education participation is high in the range of impacts found previously for mandatory 
welfare-to-work programs. 35 

The goal of these programs was not simply to augment basic skills but also to move par- 
ticipants into the labor market. Reflecting this goal, as Table 2.3 presents, the programs increased 
participation in job search (by 14 percentage points). Notice that the programs increased the pro- 

33 See Hamilton et al., 1997, for detailed participation results for the education- and employment-focused pro- 
grams in Atlanta, Grand Rapids, and Riverside; see Freedman et al., 1999, for participation results for all 11 pro- 
grams in the NEWWS Evaluation. 

All impact estimates presented in this chapter are regression-adjusted, controlling for pre-random assignment 
characteristics of sample members. 

35 Impacts on participation in adult education in mandatory programs discussed in Pauly, 1995, ranged from 4 to 
37 percentage points. 
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portion of those who participated in both adult education and job search (by 10 percentage 
points) and in both adult education and college or vocational training (by 5 percentage points). 
Few participated in these combinations of activities without the intervention of the welfare-to- 
work programs (indicated by the control group participation levels — 2 and 3 percent, respec- 
tively). 

In addition, the programs increased the length of time spent in activities. Focusing on 
adult education, notice that program group members spent an average of 176 more hours in 
this activity than their control group counterparts. (This experimental comparison includes all 
program and control group members who were surveyed; zero hours are counted for each of 
those who did not participate in adult education.) Presented in terms of months (not shown in 
the table), this means that control group members participated in adult education for an aver- 
age of 1.2 months, while program group members participated for an average of 3.7 months — 
an increase of 2.5 months. 

Considering only those who participated in adult education (a nonexperimental com- 
parison, because program group participants may have different characteristics than their con- 
trol group counterparts), it is apparent that participating program group members spent sub- 
stantially more hours in class than did control group members who were surveyed; an average 
of 488 hours compared with 388 hours, respectively. Program group participants spent 123 
more hours in ABE or GED class, 81 more hours in ESL class, and 116 more hours in high 
school completion classes. 

As mentioned, these welfare-to-work programs were mandatory; welfare recipients 
could lose part of their grant if they failed to comply with program requirements. Across the 
three programs, about one-quarter (28 percent) of program group members were sanctioned at 
some point during the two years following their entry into the programs. 36 The three programs 
have been characterized in earlier NEWWS Evaluation reports as strongly enforcing the par- 
ticipation mandate. 37 

B. Participation in Adult Education for Various Subgroups 

The preceding findings establish the ability of these education-focused programs to in- 
crease participation in adult education activities. However, it is useful to know whether these 
programs were able to do so for a range of different subgroups, including groups with low levels 
of education, high levels of barriers to participation, and little motivation to participate. This sec- 
tion examines participation in adult education for a number of interesting subgroups. 

Program effects were estimated for groups defined using the following baseline character- 
istics (that is, characteristics at the time of random assignment): 



36 The sanctioning rates differed by program. In Atlanta, 26 percent were sanctioned; in Grand Rapids, 39 per- 
cent were sanctioned; and in Riverside, 21 percent were sanctioned. See Hamilton et al., 1997, for a complete pres- 
entation of sanctioning rates and a discussion of enforcement in the three programs. (The rates presented here are 
based on client survey data. Rates presented in Hamilton et al., 1997, are based on case file data.) 

37 See Freedman et al., 1999. 
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® education levels (TALS document literacy test and CASAS math test scores 
and highest grade completed) 

° level of economic disadvantage (based on work and welfare history) 
o age of youngest child 
® self-reported family or personal problems 

o self-reported parental concerns (concerns about leaving family for work or 
school) 

® self-reported preference for school 
° self-reported depressive symptoms 

The last four subgroups were created using data from a self-administered questionnaire, com- 
pleted when sample members entered the study. 38 The creation of these subgroups is described in 
Appendix E. 39 

As Table 2.4 shows, the programs produced substantial impacts on both the level and the 
length of participation in adult education for each of the 20 subgroups examined. Participation in 
adult education was increased by 25 to 36 percentage points, and hours spent in adult education 
were increased by 74 to 237 hours. This shows that mandatory welfare-to-work programs can 
successfully increase participation in adult education for many different types of clients, includ- 
ing those facing barriers to participation or those unmotivated to enroll. 

As the first rows of the table show, control group members with low scores (Level 1-2) 
on the reading test were less likely to seek out adult education programs on their own than con- 
trol group members with high scores (Level 3-5). Nonetheless, the program effects on participa- 
tion in adult education were remarkably similar for the two subgroups (33 percentage points for 
those with low scores and 31 percentage points for those with high scores). This is also true for 
the subgroups defined using math test scores. 

Program effects on hours of participation in adult education were substantial for all four 
test-score subgroups but tended to be larger for those with low scores. The programs increased 
adult education participation by 194 hours for those with low reading scores, compared with an 
increase of 139 hours for those with high reading scores. (The “daggers” shown in the table indi- 
cate whether subgroup effects were statistically significantly different from one another.) Focus- 
ing only on participants in adult education among these subgroups (the last set of columns), note 
that the programs produced larger increases in hours of adult education for those with low read- 
ing and low math scores than for those with high scores. 

Impacts on the level and hours of participation in adult education were similar across the 
subgroups defined by highest grade completed before entering the program. Among only those 

38 See Appendix C in Hamilton and Brock, 1994, for a copy of the Private Opinion Survey instrument. 

Sample sizes vary across subgroups because some sample members are missing the relevant baseline or Private 
Opinion Survey measures. 
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who participated in adult education, the programs actually decreased the hours of participation 
for those entering at an eighth grade level or below. This probably results from three main fac- 
tors. First, control group members in this subgroup who participated in adult education partici- 
pated for more hours (586 hours) than any other control group members, so the programs had a 
high “threshold” to surmount. Second, the welfare-to-work programs generally limited the dura- 
tion of participation in education (6 to 12 months in Riverside, two years in Atlanta and Grand 
Rapids), and they possibly diverted some clients who would have continued in education into 
another activity, such as job search or vocational training. Third, many clients whom the pro- 
grams involved in adult education would have participated for zero hours without the program 
intervention; these individuals may have had substantial participation, even if it was less than the 
control group average of 586 hours. 

The programs produced larger impacts on participation for the economically more disad- 
vantaged subgroup than for the less disadvantaged subgroup. Fifty-two percent of the more dis- 
advantaged program group members participated in adult education — a 36 percentage point in- 
crease above the control group level. The corresponding increase for the less disadvantaged was 
only 29 percent. Likewise, the difference in adult education participation for the more disadvan- 
taged was 218 hours, compared with a 132-hour increase for the less disadvantaged. 

As Table 2.4 shows, the impacts for subgroups defined using reported barriers to partici- 
pation were statistically indistinguishable. However, the 25 percentage point impact for the sub- 
group with many family or personal problems is the smallest program effect among all 20 sub- 
groups. The impact for those who reported having many concerns about leaving their families to 
go to work or to participate in a job search or education activity (many parental concerns) was 35 
percentage points — one of the largest impacts among all the subgroups, and identical to the im- 
pact for the subgroup with neither barrier (35 percentage points). The hours impact for this sub- 
group (237 hours) is the largest among all 20 subgroups. This suggests that the questions asked to 
create the parental concerns scale do not capture a true barrier to participation but simply a pref- 
erence (see Appendix E for the questions included in the scale). Ideally, perhaps, these persons 
would remain at home with their families, but in the context of the 1990s welfare and economic 
environment, they are able to attend school. 

The subgroup who said that they like school and/or plan to attend school can be consid- 
ered more motivated to participate in an education activity than those who did not. One would 
expect, then, that the adult education participation level of control group members who expressed 
a preference for school would be higher; this is true (23 percent, compared with 9 percent for 
those who did not express such a preference). It would not be surprising for mandatory welfare- 
to-work programs given their incentives and penalties — to further increase education participa- 
tion for a motivated subgroup, and this was indeed the case. It would also be reasonable to as- 
sume that programs would have more trouble engaging people in education activities if they did 
not express a preference for school. Unexpectedly, though, the programs studied were also suc- 
cessful in increasing adult education participation for this subgroup — by 35 percentage points 
and by 200 hours. Program group members in this subgroup who participated in adult education 
stayed in the activity for about as long as program group members who expressed a preference 
for school (499 hours, compared with 509 hours). 
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For persons with many symptoms of depression, the programs produced smaller 
participation increases than for those with a moderate number of symptoms or few symptoms: the 
26 percentage point impact on level of participation is smaller than the impacts for the other two 
subgroups (although the difference is not statistically significant), and the 74-hour impact is 
much smaller than the impacts on hours for the other subgroups (and is the smallest hours impact 
among all the subgroups). The magnitude of this impact is a product of somewhat short stays in 
education for the program group and long stays for the control group. In fact, among participants, 
the programs decreased the length of stay in adult education for this subgroup by 163 hours. This 
decrease may reflect that some members of the program group left welfare and thus stopped at- 
tending class, that the programs limited the length of stay in education and removed those who 
were not making progress, or that the programs provided supports (perhaps mental health coun- 
seling) that allowed students to progress more rapidly through class. It is also possible that the 
programs “exempted” some clients from the participation requirement because of their mental 
condition. 

VII. Effects of the Education-Focused Programs on Key Educational 
Outcomes 

The education-focused programs in Atlanta, Grand Rapids, and Riverside substantially 
increased participation in adult education. This section discusses whether the programs also in- 
creased educational attainment (receipt of education credentials) and educational achievement 
(reading and math skills). The impacts examined in this section are based on comparisons be- 
tween average outcomes for the control group and the program group. The averages include out- 
comes both for individuals who participated in adult education and for individuals who did not. 
Thus, any effects of the education itself may be understated. Also, although treatment effects 
could have been caused by other program components that were unrelated to adult education, if a 
program improved educational outcomes — for example, by increasing receipt of a GED — it is 
unlikely that these impacts were caused by the program’s job search or training activities. 

A. Impacts on Educational Attainment 

1. Educational Attainment for the Full Sample. Table 2.5 presents the programs’ 
impacts on educational attainment using two-year survey data for 2,923 sample members. As the 
table’s first row shows, few control group members (4 percent) received a high school diploma 
or GED during the two years following random assignment. Eleven percent of the program group 
received such a credential, for a modest impact of 7 percentage points. 40 This program effect 
mostly reflects receipt of GED certificates, and its magnitude is in the range of impacts found in 
previous evaluations. 41 None of the three programs studied here increased receipt of a training 



40 Impacts varied across the three sites: the Atlanta program did not produce an impact on high school diploma or 
GED receipt; the Grand Rapids program produced an 1 1 percentage point impact; and the Riverside program pro- 
duced an 8 percentage point impact (see Freedman et al., 1999). 

“"Fourteen of 17 programs reviewed in Pauly, 1995, produced high school diploma or GED attainment impacts, 
ranging from 2 to 19 percentage points. 
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National Evaluation of Welfare-to-Work Strategies 
Table 2.5 

For Sample Members Without a High School Diploma or GEB at Random Assignment: 
Impacts ora Receipt of a High School Diploma or GED at the End of Two Years, 
Pooled Across Three Education-Focused Programs, for the Full Sample 
and by Selected Characteristics at the Time of Random Assignment 



Sample or Subgroup 


Sample 

Size 


Impact on Hours 
of Participation in 
Adult Education 


Received 

Program 

Group 


a High School Diploma or GED (%) 
Control Difference Effect 

Group (Impact) Size 3 


Full sample 


2,923 


176.2 


10.6 


3.6 


7.1 *** 


0.38 


Score on TALS document literacy 


test 

Level 1 - 2 


1,731 


193.5 


4.3 


1.1 


ttt 

3.1 *** 


0.17 


Level 3 - 5 


980 


138.8 


23.4 


7.5 


15.9 *** 


0.86 


Score on CASAS math test 
Level 1 - 2 


1,864 


187.9 


3.8 


1.0 


ttt 

2.8 *** 


0.15 


Level 3-4 


846 


148.1 


26.3 


8.6 


17.8 *** 


0.96 


Highest grade completed in school 
8 or below 


409 


145.3 


3.5 


1.2 


ttt 

2.3 


0.13 


9- 10 


1,065 


205.2 


11.2 


2.9 


8.3 *** 


0.45 


1 1 or above 


1,432 


163.6 


12.5 


4.7 


7.8 *** 


0.42 


Level of economic disadvantage 0 


More disadvantaged 


1,454 


218.4 


7.9 


2.0 


5.8 *** 


0.32 


Less disadvantaged 


1,454 


131.8 


13.3 


5.0 


8.3 *** 


0.45 


Age of youngest child 


5 or under 


1,305 


195.5 


14.1 


5.3 


8.8 *** 


0.48 


6 or over 


1,618 


158.8 


7.2 


1.7 


5.5 *** 


0.30 


Reported barriers to participation 


Many family or personal problems and 
parental concerns 


327 


171.5 


4.2 


2.6 


ttt 

1.7 


0.09 


Many family or personal problems 


223 


111.5 


5.9 


2.9 


3.0 


0.16 


Many parental concerns 


348 


237.4 


18.0 


3.6 


14.4 *** 


0.78 


Neither barrier 


1,358 


173.8 


11.7 


4.3 


7 4 *** 


0.40 


Preference for school d 

Do not like and/or do not plan 


to attend school 


873 


199.7 


8.9 


1.8 


7 j *** 


0.39 


Like and/or plan to attend school 


1,504 


170.4 


11.8 


4.9 


6.8 *** 


0.37 


Depressive symptoms d 


Many symptoms 


346 


74.4 


14.8 


5.2 


9.6 *** 


0.52 


Moderate number of symptoms 


592 


219.2 


11.6 


3.3 


8.3 *** 


0.45 


Few symptoms 


1,275 


176.2 


10.8 


3.8 


7.0 *** 


0.38 



(continued) 
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Table 2.5 (continued) 



SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test and 
CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1 . 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally in the pooled 
estimates. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members. 

Sample sizes for individual measures vary because of missing values. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between outcomes for the program and control groups. Statistical 
significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

An F-test was applied to differences among subgroups for each characteristic. Statistical significance levels are 
indicated as: f = 10 percent; ft = 5 percent; and ttt = 1 percent. 

^he effect size equals the difference (impact) divided by the standard deviation of the outcome measure for the full 
sample control group. 

b TALS scores for Riverside are based on scores earned on the Greater Avenues for Independence (GAIN) Appraisal 
literacy test and are converted to their TALS equivalent. 

c "More disadvantaged" individuals are those who did not work for pay in the year prior to random assignment and 
who received AFDC for more than two years prior to random assignment. The "less disadvantaged" category contains 
those who did not meet these criteria. 

d These subgroups are based on scales created from Private Opinion Survey data. For an explanation of these 
subgroups see Appendix E. 
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certificate or trade license (for those entering the program without a high school diploma or 
GED). 42 



2. Educational Attainment for Various Subgroups. As Table 2.5 shows, impacts 
on receipt of a high school diploma or GED certificate were produced for most, but not all, sub- 
groups. As has been found in previous studies, impacts were much larger for persons entering the 
program at higher educational achievement levels. The few subgroups without impacts were 
those who had dropped out of school very early (before beginning ninth grade) and those with 
many family or personal problems. There appears to be no clear relationship between the impact 
on hours of participation in adult education and impacts on educational attainment: subgroups 
with larger increases in the number of hours do not necessarily have larger high school diploma 
or GED impacts. (As mentioned, the relationship between length of participation and educational 
attainment is explored more extensively in Chapter 3.) 

Twenty-three percent of program group members who had high scores on the reading test 
at program entry received a diploma or GED within two years. This represents a 16 percentage 
point impact above the control mean of 8 percent. The corresponding impact for the subgroup 
with high math scores is 18 percentage points. 

Each percentage point impact on Table 2.5 is expressed also as an effect size , which 
equals the difference (impact) divided by the standard deviation of the outcome measure for the 
control group. 43 This transformation of the impacts standardizes the results for comparison across 
subgroups and across different studies and for comparison to the results presented in the next 
section, on educational achievement. 

As the table’s last column shows, the impacts on high school diploma or GED attainment 
for the two subgroups with high test scores were large: .86 of a standard deviation for the high 
reading score group and .96 of a standard deviation for the high math score group. These impacts 
are the largest of all the subgroups examined (effect sizes for the other statistically significant 
impacts range from .15 to .78 of a standard deviation). 

For the subgroups defined by the highest grade completed in school, impacts on receipt of 
a high school diploma or GED were found only for sample members who had previously com- 
pleted some high school, whether an earlier grade (9 or 10) or a later grade (11 or above). The 
subgroup that had completed eighth grade or below had no attainment impact (the 2 percentage 
point impact is not statistically significant). 

The programs did not increase receipt of a high school diploma or GED for those who re- 
ported many family or personal problems at program entry. (See the results for the first two sub- 
groups listed under reported barriers to participation. The first includes people with both many 
family or personal problems and parental concerns, and the second includes those who have 
many family or personal problems but not many parental concerns.) Interestingly, program ef- 



42 See Freedman et al., 1999, for a complete presentation of education and training credential results for these 
programs and the other NEWWS Evaluation programs. 

43 The standard deviation for the entire control group (without a high school diploma or GED) was used to calcu- 
late each subgroup’s effect size. 
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fects among subgroups defined using family or personal barriers were largest for those who re- 
ported many parental concerns. Again, as suggested in the previous section, this indicates that the 
scale is not capturing permanent barriers to successful program participation. 

Notably, reflecting the adult education participation findings, the programs produced im- 
pacts on high school diploma or GED attainment for people who said they did not like school or 
did not plan to go to school. The 7 percentage point impact for this subgroup represents .39 of a 
standard deviation. 

Despite the substantial variation in impacts on hours of participation across the three sub- 
groups defined in terms of depressive symptoms, impacts on receipt of a high school diploma or 
GED were very similar. Most notably, for those with many symptoms of depression, the some- 
what small 74-hour increase in adult education participation was accompanied by a 10 percent- 
age point impact on GED receipt. Not shown in the table, the programs also produced a small 
impact on the receipt of a trade license or certificate for the subgroup with many symptoms of 
depression. 44 This shows that some combination of the programs’ supports, incentives, and man- 
dates helped students complete education and training. 

It is important to keep in mind that these findings represent only two years of follow-up. 
In some cases — such as for those entering with low educational attainment — gains in educational 
attainment might be delayed as persons continue participating in education or improve their skills 
through work. The final report in the NEWWS Evaluation will present five-year educational at- 
tainment results for these three programs. 

As previously mentioned, the welfare-to-work programs studied here emphasized educa- 
tion and training, based on the belief that an up-front investment in the skills levels of welfare 
recipients allows them to obtain higher-paying and more secure jobs. As was just presented, the 
programs increased receipt of GED certificates; Chapter 4 examines whether receipt of a GED 
certificate led to higher earnings. 

B. Impacts on Educational Achievement 

The primary measure of educational achievement in the NEWWS Evaluation in these 
three sites is performance on the Test of Applied Literacy Skills (TALS) document literacy test, 
which was developed by the Educational Testing Service. (See Appendix D for a more detailed 
description of this test.) About two years after random assignment the test was administered to 
program and control group members who did not have a high school diploma or GED at the time 
of random assignment 45 (and to those in Riverside who had a diploma or GED but had low read- 
ing or math skills). 46 



44 Nine percent of program group members in this subgroup received a trade license or certificate, compared with 
4 percent of control group members — a 5 percentage point impact. The only other subgroup with an impact on the 
receipt of a trade license or certificate is the more disadvantaged subgroup, with a 3 percentage point impact. 

45 Administration of an achievement test as part of a survey conducted in sample members’ homes poses some 
concerns. Because the outcome of such a test has no bearing on sample members’ lives and because of distractions 
(from family members, visitors, etc.), the test’s reliability will be less than that of a test administered in a school. 

(continued) 
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Reading test scores were analyzed for sample members for whom both baseline and fol- 
low-up test score data were available (2,532 sample members). Impacts are stated both as abso- 
lute numerical effects and as effect sizes (the impact divided by the standard deviation of the test 
score calculated for all control group members in the research sample). 47 Beyond the advantages 
discussed in the previous section, stating impacts as an effect size is particularly useful when dis- 
cussing test scores because it provides a reference forjudging absolute impacts large or small. 
For example, a test score impact of 25 would be considered trivial if it were .02 of a standard de- 
viation but would be meaningful if it were .25 of a standard deviation. 

In summary, small impacts on reading test scores were found for the subgroup whose 
youngest child was 6 or older at the time of random assignment, but no evidence of reading score 
impacts was found for the sample as a whole or for any other subgroup. Subgroups for which in- 
creases in adult education hours were relatively large did not necessarily show subsequent im- 
pacts on reading scores. 

1. Reading Scores for the Full Sample. As shown in Table 2.6, the programs did not 
produce an impact on sample members’ average reading score. The programs, however, did raise 
a small proportion from Level 2 to Level 3 (accompanied by statistically insignificant reductions 
in the number of sample members scoring in Levels 4 and 5). According to test developers, the 
upward movement represents an improvement from having a severe labor market handicap to a 
level possessed by many employees in service, clerical, and other low-skill occupations. (See 
Appendix Table C.3 for program effects broken down by site.) 

2. Readimig Scores for Various Subgroups. Table 2.7 presents impacts of the three 
programs on the number of hours of adult education and reading scores for several subgroups of 
the sample of test-takers. Impacts on reading scores were found only for the subgroup of sample 
members whose youngest child was 6 or older at the time of random assignment, although the 
impact on hours of participation in adult education for these clients was not different from that 
for those with younger children. 48 This impact represents a gain of only .07 of a standard devia- 
tion — a small impact. None of the other subgroup impact estimates was statistically significantly 
different from zero. Except for the age of youngest child subgroup, the variation in reading score 
impacts across other subgroups was not statistically significant. There appears to be little correla- 
tion between impacts on average hours of participation and test scores. 

Table 2.8 presents detailed achievement impacts for those whose youngest child was 6 or 
older at the time of random assignment. This category of sample members was also administered 



While this may not directly affect the difference in test scores between program group members and control group 
members, it might make it more difficult to find statistically significant program effects. 

46 Recall that Riverside’s sample for this chapter includes people who were determined to be “in need” of educa- 
tion at random assignment (discussed in section V of this chapter). Across all sites, those with limited reading skills 
did not take the reading test and thus are not included in the educational achievement analysis. 

47 A11 test score impacts presented in this section are regression-adjusted, controlling for pre-random assignment 
characteristics and baseline test scores of sample members. 

48 When using a p-value of .1 as the cutoff to identify statistically significant impacts, one might expect to find an 
apparent impact for 1 of every 10 subgroups even if none of the groups really experienced one. Thus, this impact 
may simply represent random variation. 





National Evaluation of Welfare-to-Work Strategies 

Table 2.6 



For Sample Members Without a High School Diploma or GED at Random Assignment: 
Impacts on TALS Document Literacy Test Scores at the End of Two Years, 
Pooled Across Three Education-Focused Programs 



Measure 


Program 

Group 


Control 

Group 


Difference 

(Impact) 


Effect 

Size 3 


Average TALS document literacy test score b 


250 


249 


0.5 


0.01 


Scored at (%) 


Level 1 


26.2 


24.9 


1.2 


0.03 


Level 2 


41.2 


44.7 


-3.4 * 


-0.07 


Level 3 


26.9 


23.7 


3 3** 


0.08 


Level 4 


5.7 


6.6 


-1.0 


-0.04 


Level 5 


0.0 


0.1 


-0.1 


-0.03 


Sample size 


1,230 


1,302 







SOURCE: MDRC calculations from TALS document literacy test data. 



NOTES: About a quarter of the sample in Riverside’s education- focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, 
and thus were determined to be in need of basic education. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally in the pooled estimates. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics and baseline test scores of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between program and control groups. Statistical significance 
levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

a The effect size equals the difference (impact) divided by the standard deviation of the outcome measure 
for the full sample control group. 

^ALS = Test of Applied Literacy Skills. 
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National Evaluation of Welfare-to-Work Strategies 
Table 2.7 

For Sample Members Without a High School Diploma or GED at Random Assignment: 
Impacts ora TALS Document Literacy Test Scores at the End of Two Years, 
Pooled Across Three Education-Focused Programs, 
by Selected Characteristics at the Time of Random Assignment 



Impact on Hours TALS Document Literacy Test Score 



Subgroup 


Sample 

Size 


of Participation in 
Adult Education 


Program 

Group 


Control Difference 
Group (Impact) 


Effect 

Size a 


Score on TALS document literacy test b 


Level 1 - 2 


1,593 


193.5 


230 


228 


1.5 


0.03 


Level 3 - 5 


939 


138.8 


282 


284 


-1.3 


-0.02 


Score on CASAS math test 


Level 1 - 2 


1,720 


187.9 


234 


232 


2.1 


0.04 


Level 3-4 


812 


148.1 


279 


282 


-2.4 


-0.05 


Highest grade completed in school 


8 or below 


237 


145.3 


233 


227 


5.4 


0.10 


9- 10 


978 


205.2 


247 


248 


-1.0 


-0.02 


1 1 or above 


1,317 


163.6 


256 


255 


0.4 


0.01 


Level of economic disadvantage c 


More disadvantaged 


1,294 


218.4 


242 


242 


-0.7 


-0.01 


Less disadvantaged 


1,226 


131.8 


258 


256 


1.6 


0.03 


Age of youngest child 
5 or under 


1,155 


195.5 


257 


260 


tt 

-2.8 


-0.05 


6 or over 


1,377 


158.8 


243 


239 


3.9 * 


0.07 


Reported barriers to participation 11 

Many family or personal problems and 


parental concerns 


289 


0.0 


232 


234 


-2.0 


-0.04 


Many family or personal problems 


201 


171.5 


238 


242 


-4.3 


-0.08 


Many parental concerns 


316 


111.5 


256 


255 


1.7 


0.03 


Neither barrier 


1,260 


237.4 


257 


254 


2.8 


0.05 


Preference for school d 

Do not like and/or do not plan 


to attend school 


798 


0.0 


245 


243 


1.9 


0.02 


Like and/or plan to attend school 


1,363 


199.7 


254 


253 


0.7 


0.01 


Depressive symptoms d 


Many symptoms 


321 


0.0 


248 


248 


0.4 


0.01 


Moderate number of symptoms 


543 


74.4 


252 


253 


-1.0 


-0.02 


Few symptoms 


1,161 


219.2 


253 


252 


0.6 


0.01 



(continued) 
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Table 2.7 (continued) 



SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test 
and CASAS math test data, and Private Opinion Survey data. Calculation of impacts on TALS scores are based on all 
sample members with baseline and follow-up test score data. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally in the 
pooled estimates. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics and baseline test scores of sample members. 

Sample sizes for individual measures vary due to missing values. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between outcomes for the program and control groups. Statistical 
significance levels are indicated as: * = 10 percent; ** = 5 percent; and *** — 1 percent. 

An F-test was applied to differences among subgroups for each characteristic. Statistical significance levels are 
indicated as: | = 10 percent; ft = 5 percent; and ftt = 1 percent. 

^he effect size equals the difference (impact) divided by the standard deviation of the outcome measure for the 
full sample control group. 

hfALS scores for Riverside are based on scores earned on the Greater Avenues for Independence (GAIN) 
Appraisal literacy test and are converted to their TALS equivalent. 

c "More disadvantaged" individuals are those who did not work for pay in the year prior to random assignment 
and who received AFDC for more than two years prior to random assignment. The "less disadvantaged" category 
contains those who did not meet these criteria. 

‘bfhese subgroups are based on scales created from Private Opinion Survey data. For an explanation of these 
subgroups see discussion in Appendix E. 
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National Evaluation of Welfare-to-Work Strategies 
Table 2.8 

For Sample Members Without a High School Diploma or GED, 
Whose Youngest Child Was 6 or Older at the Time of Random Assignment: 
Impacts on TAILS Document Literacy Test and CASAS Math Test Scores 
at the End of Two Years, Pooled Across Three Education-Focused Programs 



Measure 


Program 

Group 


Control 

Group 


Difference 

(Impact) 


Effect 

Size a 


TALS document literacy and CASAS math tests 


Scored at Level 3 or above on literacy or math test (%) 


41.7 


36.4 


5.3 ** 


0.12 


Scored at Level 3 or above on literacy and math tests (%) 


18.7 


13.8 


5.0 *** 


0.15 


TALS document literacy test 


Average score 


243 


239 


3.5 * 


0.07 


Scored at (%) 


Level 1 


31.6 


32.6 


-1.0 


-0.02 


Level 2 


40.7 


45.0 


-4.4 


-0.09 


Level 3 


24.2 


17.9 


6.3 *** 


0.17 


Level 4 


3.5 


4.4 


-0.9 


-0.05 


Level 5 


0.0 


0.0 


0.0 


N/a 


Level 1 - 2 


72.3 


77.7 


-5.3 ** 


-0.13 


Level 3 - 5 


27.7 


22.3 


5.3 ** 


0.13 


CASAS math test 


Average score 


207 


206 


0.9 


0.06 


Scored at (%) 


Level 1 


24.0 


25.3 


-1.4 


-0.03 


Level 2 


43.3 


46.9 


-3.6 


-0.08 


Level 3 


23.0 


17.5 


5.5 


0.15 


Level 4 


9.7 


10.3 


-0.5 


-0.02 


Level 1 - 2 


67.2 


72.2 


-5.0 ** 


-0.12 


Level 3-4 


32.8 


27.8 


5.0 ** 


0.12 


Sample size 


662 


664 







(continued) 
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Table 2.8 (continued) 



SOURCES: MDRC calculations from TALS document literacy test and CASAS math test data. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, 
and thus were determined to be in need of basic education. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally in the pooled estimates. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics and baseline test scores of sample members. 

Rounding may cause slight discrepancies in calculating sums and differences. 

A two-tailed t-test was applied to differences between program and control groups. Statistical significance 
levels are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

N/a = not available or applicable. 

a The effect size equals the difference (impact) divided by the standard deviation of the outcome measure 
for the full sample control group. 
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the CASAS math test. 49 As shown in the table and as mentioned above, there was a small impact 
on average reading scores. Impacts for this subgroup were found mostly in Atlanta (not shown in 
the table). There was no impact on average math scores; however, there was a 6 percentage point 
increase in the number of program group members scoring at Level 3 on the math test (not statis- 
tically significant). 50 Overall, for those whose youngest child was 6 or older, there was a 5 per- 
centage point increase in the number of program group members scoring at Level 3 or above on 
both the reading and math tests. This represents a gain of .15 of a standard deviation. 51 



49 The sample presented here is slightly different from that presented in Table 2.7; Table 2.8 includes only those 
who took both the TALS document literacy test and the CASAS math test. 

50 Appendix Table D.2 describes the skills associated with the different scoring levels for the CASAS math test. 

CASAS math results are also available for 120 sample members in Grand Rapids whose youngest child was 1 
to 3 years old at the time of random assignment. There was no statistically significant impact on the TALS literacy or 
CASAS math scores for this group. However, the program did produce a 13 percentage point increase in the number 
of program group members scoring at Level 4 on the CASAS math test (not shown in any table). 



Individual Efforts and School Effects' The Payoff to 
Participation in Admit Ednicatiomi 



L ImtrodMCtiom 

A key purpose of this report is to explore and validate the model underlying education- 
focused, or Human Capital Development (HCD) programs, which emphasize skill-building 
through education as a precursor to employment. In particular, the report attempts to test the hy- 
pothesis that an initial investment in human capital will result in increases in welfare recipients’ 
skills and educational attainment, which in turn will improve both their ability to secure em- 
ployment and the wage at which they are employed. Chapter 2 began to explore this issue in three 
sites (Atlanta, Georgia; Grand Rapids, Michigan; and Riverside, California) which operated edu- 
cation-focused programs alongside more employment-focused, or Labor Force Attachment 
(LFA), programs. It described the characteristics of the adult education providers through which 
welfare recipients received adult education services, and it explored the effects of education- 
focused welfare-to-work programs on participation in these adult education activities. 

Among other things, the analysis in Chapter 2 indicated that mandatory participation in 
education-focused programs increased the amount of time spent in adult education for many dif- 
ferent groups of welfare recipients. This chapter examines the relationship between the time 
spent in adult education and welfare recipients’ educational achievement and attainment. Within 
the three sites that operated both employment- and education-focused programs, the analysis in 
this chapter focused on sample members who did not have a high school diploma or GED when 
they entered the study. 1 This chapter asks to what extent additional months spent in adult educa- 
tion are positively associated with postprogram educational outcomes among this group of wel- 
fare recipients. Specifically, we address the following questions: 

© What is the relationship between additional months of adult education and re- 
spondents’ reading skills, math skills, and the likelihood of obtaining a GED 
by the end of a two-year follow-up period? 

o How does this payoff 2 to additional months of adult education change depend- 
ing on how many months of adult education an individual has already re- 
ceived? (That is, is the payoff to additional education constant over time? 

Does it taper off after some point? Or does it follow some other pattern?) 



'This sample includes individuals from the site in Riverside who possessed a GED or high school diploma but 
who scored below a certain threshold on the reading and math tests administered at the time of random assignment or 
had limited English, and thus were determined to be in need of basic education. 

2 Xhe term payoff will be used throughout this chapter as a shorthand expression for the relationship between an 
additional month of adult education and such educational outcomes as reading and math achievement and GED re- 
ceipt. While this term implies that there is a causal relationship between participation in adult basic education and the 
education outcomes, the authors acknowledge that other factors may explain the association. This issue will be dis- 
cussed in more detail below. 



© Does the relationship between participation in adult education and individuals’ 
educational outcomes depend on whether it occurs on a voluntary basis or as 
part of a mandatory condition of their continued receipt of welfare benefits? 

© Does the relationship between participation in adult education and educational 
outcomes vary across different subgroups of welfare recipients (defined in 
terms of levels of initial educational achievement, attitudes toward additional 
education, and family and personal barriers to participation)? 

© What is the relationship between characteristics of adult education providers 
and the payoff to welfare recipients who participate in adult education? 

II. Summary of the Findings 

• Our analyses suggest a positive relationship between time spent in adult 
education and educational outcomes. However, this relationship was not 
linear and varied for different outcomes and subgroups. 

Analyses presented in this chapter reveal that welfare recipients’ educational outcomes 
were positively associated with time spent in adult education. This was true for measures of 
postprogram literacy and math achievement as well as GED receipt. However, this relationship 
changed as individuals spent more time in adult education programs. 

There was no relationship between additional months of adult education and reading 
achievement during the first year of participation, but after the first year additional months of 
adult education were associated with significant increases in reading scores. 

The relationship between math scores and participation in adult education followed almost 
the opposite pattern: there was no significant relationship between additional months of adult edu- 
cation after the first six months of participation, but there was a relationship during the first six 
months of participation. The magnitude of the relationship between time spent in adult education 
and literacy and math achievement outcomes was comparable to or greater than the relationship be- 
tween these outcomes and time spent in formal education before random assignment. 

The relationship between GED receipt and time spent in adult education followed a pat- 
tern similar to that of math scores. Early on, additional months of adult education increased GED 
receipt, but after six months of participation additional months were associated with lower levels 
of GED receipt. 

• The relationship between test scores and months in adult education was 
similar for program group members and for control group members. 

Both program and control group members experienced similar relationships between par- 
ticipation in adult education and educational outcomes. Thus, the payoff from adult education in 
terms of improved literacy and math skills was not related to whether participation was voluntary 
or a mandatory condition for the receipt or welfare benefits. 
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o Tine relationship between months in adult education and GED receipt 
was stronger for program group members. 

Those mandated to participate in adult education experienced a stronger relationship be- 
tween participation and GED receipt than those who volunteered. 

® The relationship between GED receipt and time spent in adult education 
was significantly stronger for those experiencing a more intensive pro- 
gram. 

Those who spent more than 20 hours a week in adult education experienced a stronger 
relationship between participation and GED receipt. This also appeared to be true of the link be- 
tween participation and test scores, but that difference was not statistically significant. 

® The relationship between GED receipt and time spent in adult education 
was mostly limited to those entering adult education at the ninth grade 
level or higher. 

Among sample members who had completed eight or fewer years of formal education 
when they entered the study, there was almost no relationship between months of participation in 
adult education and the receipt of a GED or high school diploma. On the other hand, among in- 
dividuals with more than eight years of formal education, there was a significantly larger and 
positive association between GED receipt and months in adult education. 

» Sample members with lower skill levels were not likely to receive a GED. 
However, longer participation in adult education increased the likelihood 
that they would. 

Individuals with lower levels of academic achievement (as measured by test scores) had a 
lower probability of receiving their GED than individuals with higher skills. However, after six 
months of participation they experienced a stronger relationship between GED receipt and total 
months spent in adult education. 

® Barriers to participation and motivation to participate in adult education 
did not affect the relationship between participation and subsequent edu- 
cational outcomes. 

The relationship between educational outcomes and months of participation was unre- 
lated to the presence of family and personal problems, to whether or not sample members had 
young children, or to whether or not they preferred attending school as opposed to some other 
activity. Moreover, the payoff to adult education in terms of improved literacy skills also did not 
vary significantly with individuals’ initial academic preparation. 

• Teachers’ experience and education levels appeared to enhance the bene- 
fits of participation in adult education as far as literacy and math scores 
were concerned. It did not similarly affect GED receipt. 

The relationship between months of participation and test score outcomes was stronger for 
sample members who attended classes at adult education providers where teachers were more ex- 
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perienced and were more likely to possess advanced degrees. However, the relationship between 
GED receipt and total months spent in adult education was not related to the number of staff with 
advanced degrees and was actually somewhat negatively associated with teachers’ experience. 

The remainder of the chapter proceeds as follows. Section III briefly reviews the data 
sources for this chapter and describes the welfare recipients who participated in adult education. 
Section IV outlines the analytic approach to answering the questions explored in this chapter. 
Section V presents the analysis of the relationship between educational outcomes and time spent 
in adult education, and section VI offers conclusions. 



IIS. Samples amid Bala Sources 



The sample for this analysis is essentially the same as the sample used in Chapter 2, 
which includes data from the three programs in Atlanta, Grand Rapids, and Riverside. Within 
these programs, the analysis in this chapter is based on welfare recipients who were either part of 
a mandatory education-focused program or those who were randomly assigned to a control 
group. Control group members were excluded from the mandatory welfare-to-work program but 
could seek out services in the community on their own. Within this three-site sample, we focused 
on those without a high school diploma or GED, including in Riverside a group who did have a 
high school diploma or GED but who scored below a certain level on the reading and math tests 
administered at the time of random assignment. 

A variety of data was collected covering these welfare recipients’ demographic characteris- 
tics, attitudes, prior educational achievement and attainment, and personal circumstances. These 
data include reading and math achievement tests administered both at the outset of the study and 
two years later. For added detail on data sources, please refer to Chapter 2 and Appendix A. 

B. Educatiom Provider Data 

As mentioned previously in this report, the NEWWS Evaluation extended data collection 
to adult education providers serving sample members in the three sites studied. In particular, field 
research was conducted — including interviews with administrators and teachers — in order to 
gather information on such provider characteristics as teachers’ experience and training, curricu- 
lum and teaching methods, and program organization, among others. These characteristics were 
discussed in Chapter 2 and will be used here to explore whether provider characteristics influ- 
ence the payoff to participation in adult education. 

The agencies covered by the interviews and surveys were the largest providers of adult 
education in the three sites’ welfare-to-work programs. We completed interviews with adminis- 
trators at 33 agencies that served between 20 and 34 percent of sample members across the three 
sites. Although these providers interacted with only a minority of sample members, they served 
the majority of individuals who participated in adult education (from 74 percent in Atlanta to 89 
percent in Riverside). Nevertheless, the fact that we did not have provider data for all sample 
members who participated in adult education did constrain the analysis and limited our ability to 
estimate relationships between provider characteristics and educational outcomes. 
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The primary focus of this chapter is an analysis of the relationship between the extent of 
participation in adult education and postprogram educational outcomes. Therefore, while we 
compared educational outcomes of those who reported any participation in adult education with 
those who did not, the primary sample for this chapter is the group of clients who actually par- 
ticipated in adult education activities. The analysis of the correlation between provider character- 
istics and the relationships between postprogram educational outcomes and total months spent in 
education will be based on a smaller sample of individuals who not only participated in adult 
education but also attended classes at education providers for which we have data on teacher and 
school characteristics. 

Table 3.1 compares baseline characteristics of the sample as a whole with those of sample 
members who participated in adult education activities and attended classes at one of the provid- 
ers for which we possess data. While the table indicates the existence of some differences across 
these groups, overall the groups appear to be fairly similar. In particular, respondents who par- 
ticipated in adult education activities were slightly more likely to have children age 5 or younger, 
were less likely to be black, and were more likely to be neither black nor white. They also were 
about a year and a half younger at the time of random assignment, had somewhat higher test 
scores, and were substantially more likely to have participated in some kind of education or train- 
ing activity in the past 12 months. 

IV. Analytic Approach 

The basic approach to the analysis in this chapter is a type of “production function” ap- 
proach that has been applied successfully in estimating the effects of education on academic 
achievement in a number of studies. 3 This approach hypothesizes that educational gains are a 
function of a variety of factors, including students’ background characteristics; their prior aca- 
demic achievement; and the nature, quality, and quantity of the educational resources to which 
they are exposed. This can be summarized as 

4, = /(4,-. > x i > Att > > s i ) > where 

A h represents academic achievement at the time of the follow-up survey of student i, at adult 
education provider j, as measured by an achievement test score or some other measure; 

A jt _, represents the baseline level of academic achievement or prior preparation of student i, 
again measured by an achievement test or some other measure; 4 

(continued) 



3 See, for example, Ferguson, 1991; Ferguson and Ladd, 1996; Hanushek, 1972, 1986; and Mumane, 1981. 

4 Another approach to modeling students’ achievement outcomes would be to use the students’ achievement 

gains as the dependent variable, that is, (A it — 4 ,- 1 ) = ^(4 > AttjSj) . Provided that the coefficient associated 

with Af is equal to 1 , this model is equivalent to the one illustrated in the text. However, we consider it preferable 

to allow the coefficient on baseline achievement to be estimated empirically rather than artificially restricting this 
coefficient to equal 1. 
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National Evaluation of Welfare-to-Work Strategies 
Table 3.1 



For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Means of Background Characteristics for Respondents with Control Variables 







Nonparticipants 


ABE Participants 


Background Characteristic 


Full Sample 


or No Provider Data 


with Provider Data 



Reading and math skills at random 
assignment 



TALS document literacy test score 


264.35 


262.53 


270.70 *** 




(45.03) 


(45.12) 


(44.18) 


CASAS math test score 


208.94 


208.41 


210.79 *** 




(13.41) 


(13.21) 


(13.94) 


Education status at random assignment 


Highest grade completed 


10.33 


10.38 


10.17 *** 




(1.31) 


(1.31) 


(1.27) 


Enrolled in education or training 
activity in 1 2 months prior to 


random assignment 


0.18 


0.15 


0.26 *** 




(0.38) 


(0.36) 


(0.44) 


Attitudes and opinions at random 
assignment 1 * 


Family or Personal Problems scale 


2.10 


2.12 


2.02 *** 




(0.65) 


(0.66) 


(0.61) 


Parental Concerns scale 


2.15 


2.15 


2.13 




(0.56) 


(0.55) 


(0.58) 


Preference for School scale 


2.25 


2.28 


2.14 *** 




(0.49) 


(0.49) 


(0.47) 


Depressive Symptoms scale 


2.13 


2.16 


2.06 ** 




(0.93) 


(0.93) 


(0.91) 


Mastery scale 


2.22 


2.22 


2.21 




(0.58) 


(0.57) 


(0.58) 


Family status at random assignment 


Ever married 


0.46 


0.45 


0.49 




(0.50) 


(0.50) 


(0.50) 


2 children 


0.34 


0.33 


0.35 




(0.47) 


(0.47) 


(0.48) 


3 or more children 


0.31 


0.31 


0.32 




(0.46) 


(0.46) 


(0.47) 


Any child 5 years old or younger 


0.46 


0.45 


0.51 ** 




(0.50) 


(0.50) 


(0.50) 



(continued) 
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Table 3.1 (continued) 



Background Characteristic 


Full Sample 


Nonparticipants 
or No Provider Data 


ABE Participants 
with Provider Data 


Public assistance and labor force 








status 








Cumulative years on public assistance 


5.10 


5.11 


5.05 




(1.60) 


(1.60) 


(1.61) 


First spell of AFDC receipt 


0.13 


0.12 


0.15 




(0.33) 


(0.32) 


(0.35) 


Any paid work in year prior to 








random assignment 


1,072.44 


1,068.45 


1,086.38 




2,549.20 


2,538.18 


2,590.22 


Demographic characteristics 








Female 


0.97 


0.96 


0.98 * 




(0.18) 


(0.19) 


(0.14) 


Age 


31.06 


31.41 


29.84 *** 




(6.84) 


(6.94) 


(6.32) 


Black 


0.57 


0.60 


0.48 *** 




(0.50) 


(0.49) 


(0.50) 


Not black or white 


0.14 


0.13 


0.18 ** 




(0.35) 


(0.34) 


(0.39) 


Sample size 


1,926 


1,497 


429 



SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test 
and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 



NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

A two-tailed t-test was applied to the means listed above, comparing participants with provider data to those who 
were either nonparticipants or for whom provider data were not available. Statistical significance levels are indicated 
as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

Numbers shown in parentheses are the standard deviations associated with the coefficient shown above. 

"The calculations in this table include only sample members who had both baseline and follow-up test score data 
in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult education within 
2 years of random assignment for more than 1 month but less than 25 months. Individuals who participated for 25 
months or more were excluded due to small sample sizes. 

b These subgroups are based on scales created from Private Opinion Survey data. For an explanation of these 
subgroups see Appendix E. 




4 



- 67 - 



X . represents the background characteristics of student i, such as his or her initial level of 
motivation and the supports or barriers to academic achievement which exist in the envi- 
ronment; 

5, represents the characteristics of the training provider or educational institution which 
served student i; and 

Att j is a measure of the level of participation of student i in the education activities — in this 
case, months of attendance in adult education. 

The analysis in this chapter focuses on variation in postprogram educational outcomes 
across individuals with different lengths of stay in adult education, controlling for factors such as 
motivation and other background characteristics. Among other things, this analysis will allow us 
to estimate the relationship between students’ improvements in measured cognitive skills and the 
duration of their participation in adult education activities, controlling for prior achievement and 
the background characteristics discussed above. The basic model can be written as 

At = A + A^tt, + A,-i +/3 i X i +r ij . s (1 ) 

At this level of analysis, we are primarily interested in the relationship between postprogram 
achievement and total months of participation in the adult education activities during the two- 
year follow-up period. This is represented by /?, , the estimated payoff to an additional month in 
adult education. 

This estimate draws on variation across individuals who spent different amounts of time 
in adult education activities. It is important to note that while our analysis controls for observed 
characteristics such as initial academic achievement and attitudes toward school, it is possible 
that individuals who spent more months in adult education were different from those who spent 
fewer months, in ways that were not directly observed and could not be controlled for in the 
analysis. For example, individuals who spent more time in adult education might have been more 
motivated to improve their academic skills than those who spent less time in adult education. To 
the extent that such differences existed and also directly affected postprogram outcomes, the 
analysis may not have accurately captured the causal relationship between educational outcomes 
and time spent in adult education. On the other hand, it is not clear how unobserved differences 
in factors such as motivation would have translated into different postprogram achievement lev- 
els except through additional participation in adult education activities. 

One of the questions we intended to explore is whether or not the relationship between 
time spent in adult education and postprogram educational outcomes was affected by individual 



5 In this equation, the notation of error term r represents the assumption that the error in this equation consists of 

an individual error term and a provider-level error term. In other words, the error is specific to individual i at adult 
education provider j. In order to account for this, statistical estimation procedures included a series of dichotomous 
variables representing each adult education provider. (These variables are not shown in the equation presented 
above.) The resulting estimates, known also as “fixed effects” estimates, are net of any direct effects of unobserved 
provider characteristics on postprogram educational attainment and achievement from the error term in the equation; 
they therefore yield correctly estimated coefficients and standard errors. 
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background characteristics such as prior educational achievement, family and personal problems, 
or the presence of young children. In other words, does the payoff to additional months of adult 
education differ across subgroups of welfare recipients with different background characteristics? 
We addressed this question by adding “interactions” between individual characteristics and 
months spent in adult education to the earlier model, as shown: 

4m = Po + R Att ij + A4/.m + & X v + PA X 9 Att t ) + r tj . ( 2 ) 

so that we could estimate the average payoff to additional months of adult education (represented 
by /?, ) at the same time we estimated the differences in the payoff to an additional month of adult 
education across individuals with different background characteristics (represented by # ). 

The analysis in this chapter also seeks to ascertain how the relationship between educa- 
tional outcomes and time spent in adult education varies with the characteristics of the adult edu- 
cation providers. 6 For example, is the payoff to months in adult education significantly higher at 
adult education providers with smaller classes? In order to answer this question we estimated 
models such as the following: 

A„ = # + ftAtt, + + &X l + &(X i Att i ) + AiS'Att,) + r y . (3) 

In this equation S, represents a provider characteristic — for example, the class size — for a pro- 
vider at which individual i attended classes. Therefore# represents the difference in the payoff 
to adult education across providers with different characteristics. 7 In other words, this coefficient 
estimates the relationship between provider characteristics such as class size and the payoff to an 
additional month spent in adult education. 

V. Relationship Between Educational Attainment and Achievement 
and Participation in Basic Education 

A. Exploratory Analysis 

1. Math Achievement. As pointed out above, our primary analytical focus was to es- 
timate the payoff to time spent in adult education, that is, the relationship between individuals’ 
postprogram educational outcomes and the amount of time they spent in adult education activi- 
ties during the two-year follow-up period. As a first look at this relationship, Figures 3.1, 3.2, and 
3.3 plot the average postprogram CASAS math score, TALS reading score, and level of GED 



6 While this basic formulation is quite common, it does not necessarily take into account or take advantage of the 
multilevel nature of the data. In particular, a hierarchical model — of the kind described by Raudenbush and Bryk, 
1 992 — may be the most appropriate way to approach the estimation of the education production function described 
above. Unfortunately, we had insufficient data to carry out such an approach and, instead, relied on interaction terms 
between provider- and site-level variables to identify how these relationships varied with the characteristics of the 
adult education providers. 

7 As mentioned above, these estimates included specific intercepts (fixed effects) for each provider. Therefore, it 
was not possible to estimate differences in postprogram outcomes that resulted from direct effects of provider char- 
acteristics. Only the interaction between provider characteristics and the payoff to months of adult education was 
estimable. 
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receipt (respectively) against two-month increments of participation in adult education. 8 The first 
point on Figure 3.1 reports average postprogram math achievement among individuals who re- 
ported no participation in adult education; the second point represents the average level of 
achievement among individuals with 1 or 2 months of participation; the third point on the graph 
represents average postprogram math achievement among individuals with 3 or 4 months of par- 
ticipation; and so on. 9 

The shape of Figure 3.1 suggests that the relationship between adult education and post- 
program math achievement is flat for the first few months and is somewhat steeper between 4 
and 8 months. In particular, the average math score among welfare recipients in the sample who 
reported no participation in adult education was about 208 points — about the same as those who 
reported either 1 to 2, or 3 to 4 months of participation. Among individuals who reported 7 to 8 
months of participation, the score climbed to about 215 points. The CASAS has a standard devia- 
tion of approximately 14 points. Therefore, the 7-point difference between the test scores of re- 
spondents with 3 to 4 months of adult education and those with 7 to 8 months of adult educa- 
tion — translated into the “effect size” metric — can be thought of as a difference of approximately 
.50 of a standard deviation. 

Figure 3.1 also shows that the payoff to months of adult education in terms of postpro- 
gram math scores begins to trend downward after this point, and it rises again a few months later. 
On the whole, however, the relationship after this point appears to be basically flat. Therefore 
Figure 3.1 suggests that while there is a slight delay in the payoff to additional months of adult 
education, the effect of adult education on postprogram math achievement is concentrated during 
the early months of participation and declines strongly in the second half of the first year of pro- 
gram attendance. 

2. Reading Achievement. Figure 3.2 suggests a somewhat different pattern in the re- 
lationship between time spent in adult education and postprogram reading skills. During the first 
year of attendance, the relationship between months spent in adult education and subsequent 
reading scores is weak or insubstantial. However, after one year the relationship appears to turn 
upward. Specifically, the average postprogram TALS document literacy test score among re- 
spondents with no participation in adult education was about 249 points. Among those with 1 1 to 
12 months of participation in adult education, this score was essentially unchanged (247 points). 
However, for those who reported 17 to 18 months of participation in education, the average 
TALS reading score was 274 points — approximately 27 points higher. Inasmuch as the standard 
deviation of the TALS among respondents in our sample was approximately 49, this difference 
translates into an effect size of .55 of a standard deviation. 



Specifically, follow-up TALS scores, CASAS scores, and GED receipt were regressed on a series of dichoto- 
mous variables representing eight-week increments of participation in adult education and a series of individual- 
level control variables for background characteristics. These regression estimates were combined with sample aver- 
ages in order to generate predicted values for sample members with average characteristics. 

The graphs of reading and math achievement are scaled so the extreme values on the vertical axes each repre- 
sent points two standard deviations above and below the mean, respectively. Thus, rather than representing the total 
universe of possible test scores, they represent 95 percent of the distribution of test scores that was actually observed 
in our sample. 
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National Evaluation of Weifare-to- Work Strategies 



Figure 3.1 

For Sample Members Without a High School Diploma or GED at Random Assignment: 
Predicted Score on CASAS Math Test Two Years 
After Random Assignment, by Months of Participation in Adult Education 




SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test and 
CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or GED, 
but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were determined to 
be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

The y-axis spans approximately plus or minus 2 standard deviations from the mean TALS document literacy score. 
‘‘The calculations in this table include only sample members who had both baseline and follow-up test score data in the 
Atlanta, Grand Rapids, and Riverside education-focused programs. 
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National Evaluation of Wei fare-to- Work Strategies 
Figure 3.2 

For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Predicted Score on TALS Document Literacy Test Two Years 
After Random Assignment, by Months of Participation in Adult Education 




SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test and 
CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

The y-axis spans approximately plus or minus 2 standard deviations from the mean TALS document literacy score. 

a The calculations in this table include only sample members who had both baseline and follow-up test score data in 
the Atlanta, Grand Rapids, and Riverside education-focused programs. 
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This suggests several possibilities. Among the first that comes to mind is that those who 
remained in adult education for more than 12 months were somehow different from the partici- 
pants who left adult education before that point. One might argue that, even controlling for ob- 
servable background characteristics, students who remained in adult education were likely to be 
more motivated and better prepared and thus were more likely to score highly on postprogram 
achievement tests, regardless of their participation in adult education and regardless of their pre- 
program test. However, if the differences in test scores were a function of this sort of “selection 
bias,” we might expect that each additional month of adult education would be associated with 
better preparation and higher levels of motivation. The fact that there was no observed relation- 
ship between months in adult education and educational achievement for the first 12 months of 
participation suggests that this was not the case. Moreover, if those who stayed in adult education 
beyond a year had different underlying unobserved characteristics than their counterparts who did 
not, we might have expected a shift in the line, rather than the substantial change in the slope of 
the line shown in Figure 3. 2. 10 

Finally, to the extent that those who remained in adult education between a year and 18 
months were somehow “different” or more motivated than other students, we would expect to see 
higher levels of both reading and math achievement for these individuals, not just higher literacy 
levels. As Figure 3.1 revealed, math scores did not increase after the first 12 months. Therefore, it is 
difficult to argue that unmeasured underlying characteristics accounted for the apparent effects of 
long-term participation in literacy scores. If this were the case, these characteristics would somehow 
have had to affect measured literacy without affecting math scores as well. 

Therefore, rather than merely reflecting the selection of a group of individuals who were 
inherently different from their counterparts with fewer months of participation, the pattern in 
Figure 3.2 suggests that effects of participation in adult education on literacy may have been 
weak at first but may have increased substantially after a year of participation in adult education. 
This suggests that a threshold level of services was needed to substantially improve the literacy 
skills of this sample. It may take a great deal of consistent effort at establishing a basic founda- 
tion before participation in a literacy program has any substantial benefit or a meaningful effect 
on skills measured by the TALS. On the other hand, math achievement may improve at a rela- 
tively early stage of participation in adult education, but it does not appear to improve after a year 
in the program. This suggests that while basic math skills may have been relatively easy to im- 
prove, students’ either reached a plateau in their progression toward better math skills or the pro- 
grams shifted their focus away from math skills and toward literacy skills as students mastered 
basic math and spent more time in the program. 

Finally, Figure 3.2 indicates a downward trend in reading test scores after 18 months in 
adult education. This pattern does not differ substantially from the pattern observed in Figure 3.1 
regarding math scores, as well as the pattern which will be illustrated below regarding GED re- 
ceipt. Unless we believe that, all other things being held constant, adult education could actually 



l0 It is also possible that the length of time between follow-up testing and school exit may have had a negative ef- - 
feet on test scores. If, for example, the test scores of students who exited one program quickly were biased down- 
ward by taking the test several months later, the estimated payoff to adult education among those individuals might 
also be biased downward. 
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depress educational achievement, it is reasonable to conclude that the apparent downward trend 
near the end of the observation period represents a kind of negative “selection bias.” In particu- 
lar, individuals with long spells of participation may have been unable to progress in their adult 
education classes, because of lack of motivation, learning disorders, or other barriers. 

These findings may help to explain why the estimates from Chapter 2 suggest that as- 
signment to an education-focused program increased participation in adult education without in- 
creasing postprogram test scores. In particular, while this analysis suggests a relationship be- 
tween time spent in adult education and postprogram reading and math scores, it also suggests 
that this effect was delayed and subject to reaching a given threshold level of participation that 
relatively few sample members reached. In particular, there does not appear to have been a sub- 
stantial relationship between postprogram TALS literacy scores and additional months spent in 
adult education during the first 12 months. While the payoff to adult education in terms of math 
scores appears to occur sooner than for reading, there still was a period of several months during 
which, according to Figure 3.1, the relationship between postprogram math scores and months of 
adult education also appeared to be relatively flat. The impacts on participation reported in Chap- 
ter 2 amounted to about two and a half months of participation in adult education, and most of 
this seemed to occur during the first few months of respondents’ participation in the program. 
Therefore, it is not surprising that these did not translate into statistically significant impacts on 
student achievement. 

3. GED Receipt. Figure 3.3 illustrates what might be considered the “credential” effect 
of time spent in adult education. It shows the relationship between the number of months spent in 
adult education and receipt of a GED during the follow-up period. The relationship during the first 
year or so followed a somewhat “bell-shaped” pattern. In particular, there was a steep positive rela- 
tionship at the outset, followed by a range in which the relationship between participation and GED 
receipt was relatively flat but declining. This, in turn, was followed by a range of more steeply de- 
clining returns nearer the end of the first year, which was followed by another, shorter bell-shaped 
pattern. Specifically, about 2 percent of those who did not participate in adult education received a 
GED, while approximately 15 percent of those who spent 1 or 2 months in adult education received 
a GED. It seems unlikely that this initial difference of 13 percentage points was entirely the result of 
the education received by participants. Instead, it would appear that underlying differences between 
those who participated in education and those who did not accounted for at least part of the effect. 
Also, it is plausible that a number of short-term participants in adult education entered the program, 
discovered that they were close to getting a GED, took the test, passed, and left. Those who never 
participated might not have had such an opportunity. 

The differences between 1 to 2 months and 5 to 6 months of participation in adult educa- 
tion indicate that the relationship flattens out after the first month but is still positive. In particu- 
lar, 24 percent of those who reported 5 to 6 total months of participation in adult education dur- 
ing the follow-up period reported receiving their GED or high school diploma during this time — 
an increase of 9 percentage points. The line slopes downward to 18 percent by about 12 months, 
and then it experiences a steep dip to 5 percent at 13 to 14 months. Next, the likelihood of obtain- 
ing a GED during the follow-up period experiences a relatively sharp increase between 13 to 14 
months and 1 7 to 1 8 months, and it drops off substantially after that. 
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National Evaluation of Welfare-to-Work Strategies 
Figure 3.3 

For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Predicted Likelihood of High School Diploma or GED Receipt Within Two Years 
After Random Assignment, by Months of Participation in Adult Education 




SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test and 
CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

The y-axis spans approximately plus or minus 2 standard deviations from the mean TALS document literacy score. 
“The calculations in this table include only sample members who had both baseline and follow-up test score data in 
the Atlanta, Grand Rapids, and Riverside education-focused programs. 
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The interpretation of this pattern is problematic, because the relationship between GED 
receipt and participation in adult education can be reciprocal: adult education prepared sample 
members for the GED, increasing their likelihood of passing the test; in most cases, however, 
GED receipt also would have ended participation in adult education, thereby reducing time spent 
in it. Especially for sample members who spent long periods in adult education, failure to pass 
the GED test may have caused their participation spells to be lengthened. This explanation for 
the negative relationship between long-term participation and reduced GED receipt is more plau- 
sible than an explanation which posits that more education would make one less likely to take or 
pass a GED test. 

B. Relationship Between Postprogram Educational Outcomes and Time Spent 
in Adult Education 

The previous section described the observed patterns in the relationship between reading 
scores, math scores, GED receipt, and time spent in adult education. The analysis presented be- 
low built on this evidence in two ways. First, we estimated the effect of additional months of 
adult education on educational outcomes. Second, we evaluated how this payoff to additional 
education varied with the amount of adult education already accumulated. In other words, was 
the payoff to an additional month of education different in the first month of participation than in 
the sixth or twelfth month? Finally, we explored the extent to which the estimated payoff to adult 
education varied across different groups of welfare recipients, depending on their background 
characteristics, personal circumstances, and other factors. 

1. Reading Achievement. Table 3.2 reports regression estimates of the relationship be- 
tween additional months of adult education and students’ postprogram scores on the Test of Ap- 
plied Literacy Skills (TALS) document literacy test, controlling for baseline academic achievement 
and other background characteristics. The coefficients in this table represent the difference in post- 
program TALS scores associated with a one-unit increase in each variable, all other variables in the 
table being equal. 

The equation whose coefficients are presented in column 1 of Table 3.2 includes a vari- 
able indicating whether or not the respondent reported at least one month of participation in adult 
education during the follow-up period and, beyond that point, a variable equal to the number of 
additional months of adult education. The coefficient associated with the variable for ever par- 
ticipating represents the difference between those who spent any time in adult education during 
the two-year follow-up period and those who received no adult education at all. The coefficient 
associated with additional months of participation represents the average difference per month in 
postprogram test scores across adult education participants who reported different amounts of 
participation during the follow-up period. This might be interpreted as the average payoff to a 
month of adult education. 

The estimates shown in column 1 indicate the existence of a significant relationship be- 
tween postprogram literacy skills and total months in adult education. In particular, the estimates 
indicate that, among individuals with at least one month of adult education, an additional month 
of adult education is associated with a difference of .57 points on the postprogram TALS reading 
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National Evaluation of Welfare-to-Work Strategies 
Table 3.2 



For Sample Members Without a High School Diploma or GED at Random Assignment: 3 
Relationship Between Participation in Adult Education Activities and 
Two-Year TALS Document Literacy Scores 



Coefficient 


Independent Variable 


Model 1 




Model 2 


Participation in adult education within 
2 years of random assignment 








1 month of participation 


3.35 




3.91 




( 6 . 11 ) 




(6.31) 


Total months, beyond 1 


0.57 


** 


- 




(0.23) 




- 


Months beyond 1 , through 6 


- 




1.10 








(0.85) 


Months beyond 6 , through 12 


- 




-0.96 








( 1 . 00 ) 


Months beyond 1 2, through 1 8 


- 




3.30 * 




- 




(1.75) 


More than 1 8 months 


- 




-5.25 




- 




(9.89) 


Reading and math skills at random 
assignment 








TALS document literacy test score 


0.52 


*** 


0.52 *** 




(0.03) 




(0.03) 


CASAS math test score 


0.47 




0 47 *** 




(0.08) 




(0.08) 


Education status at random assignment 








Highest grade completed 


1.89 


*** 


j *** 




(0.70) 




(0.70) 


Enrolled in education or training 








activity in 12 months prior to 
random assignment 


1.56 




1.70 




( 2 . 12 ) 




( 2 . 12 ) 


Attitudes and opinions at random 
assignment 11 








Family or Personal Problems scale 


0.62 




0.56 




(1.74) 




(1.74) 


Parental Concerns scale 


- 2.00 




-1.87 




( 2 . 02 ) 




( 2 . 02 ) 


Preference for School scale 


0.38 




0.48 




(1.81) 




(1.81) 


Depressive Symptoms scale 


-0.03 




-0.15 




(0.93) 




(0.93) 


Mastery scale 


-3.91 


#* 


-3.83 ** 




(1.73) 




(1.73) 



(continued) 
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Table 3.2 (continued) 





Coefficient 




Independent Variable 


Model 1 


Model 2 



Family status at random assignment 



Ever married 


-0.37 


-0.24 




(1.98) 


(1.98) 


2 children 


-0.16 


0.01 




(2.04) 


(2.04) 


3 or more children 


2.20 


2.15 




(2.24) 


(2.24) 


Any child 5 years old or younger 


-1.66 


-1.73 


Public assistance and labor force 
status at random assignment 


(2.01) 


(2.01) 


Cumulative years on public assistance 


-0.98 


-0.94 




(0.64) 


(0.64) 


First spell of AFDC receipt 


2.60 


2.54 


Any paid work in year prior to random 


(2.64) 


(2.64) 


assignment 


0.00 


0.00 


Demographic characteristics 


(0.00) 


(0.00) 


Female 


5.00 


5.03 




(5.02) 


(5.02) 


Age 


-0.60 *** 


-0.60 *** 


Black 


(0.16) 


(0.16) 


-13.02 *** 


-12.90 *** 




(2.41) 


(2.41) 


Not black or white 


-1.89 


-2.03 




(2.86) 


(2.86) 


R-squared 


0.47 


0.47 


Sample size 


1,955 


1,955 



(continued) 
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SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test 
and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1 . 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * = 10 
percent; ** = 5 percent; and *** = 1 percent. 

Numbers shown in parentheses are the standard errors associated with the coefficient shown above. 

^he calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult education 
within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who participated 
for 2 1 months or more were excluded due to small sample sizes. 

b These subgroups are based on scales created from Private Opinion Survey data. For an explanation of these 
subgroups see Appendix E. 
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test. Moreover, this difference is statistically significant at the 5 percent level." Taken by itself, 
the average payoff to an additional month of adult education might seem relatively small com- 
pared to the distribution of test scores in our sample. In particular, in terms of effect size, the es- 
timated effect is only .012 of a standard deviation. 

However, after six months, this effect would translate into a difference of 3.42 points, or 
.07 of a standard deviation. As a basis for comparison, we estimated the relationship between 
sample members’ baseline reading and math test scores and the highest grade completed at the 
time of random assignment. These estimates suggest that, in this sample, an additional year of 
formal education was associated with a difference of 8.3 points, or .17 of a standard deviation, on 
the TALS. This suggests that the average literacy payoff to six months in adult education was 
slightly more than 40 percent of the payoff to an average academic year of the schooling the indi- 
viduals in our sample received prior to entering the Job Opportunities and Basic Skills Training 
(JOBS) program. 

As the graphs presented earlier suggested, the payoff to additional months of adult educa- 
tion appeared to vary according to sample members’ length of stay in adult education. For exam- 
ple, the difference in literacy achievement between individuals with 13 as opposed to 12 months 
of adult education might be greater than the difference between individuals with 1 and 2 months 
of adult education. In order to account for this, we estimated a different version of the equation 
linking participation in adult education and postprogram test scores, allowing the payoff to time 
spent in adult education to vary across six-month intervals of participation. 12 The results are pre- 
sented in column 2 of Table 3.2. 

The estimates in column 2 reveal a pattern that is consistent with the one first seen in 
Figure 3.2. It appears that the association between postprogram reading scores and additional 
months spent in adult education during the first year was not statistically distinguishable from 
zero. However, the estimated effect of an additional month of participation between months 13 
and 18 was both statistically significant and substantively important. In particular, each addi- 
tional month after a year of participation was associated with a 3.3 point gain in the postprogram 
reading scores (for an effect size of .067). Based on this estimate, the payoff to six additional 
months of adult education after the first year would be approximately 19.8 points, or .40 of a 
standard deviation, on the TALS document literacy test. 

It may be useful once again to compare these differences with the differences in baseline 
literacy scores associated with years of formal education before participation in the JOBS pro- 
gram. As mentioned above, a year of formal education before the beginning of the program was 
associated with a difference of 8.3 points, or .17 of a standard deviation, in baseline TALS 
scores. If we assume that each academic year of formal education consisted of approximately 9 
months of schooling, this estimate suggests that 18 months of formal education was associated 



"in other words, the chances that the data would yield an estimate of this magnitude, given that the true relation- 
ship is zero, are less than 5 percent. 

l2 The estimate in column 2 of the table also includes an indicator variable for whether or not an individual spent 
more than 1 8 months in adult education. Because few individuals participated for that long, it was not possible to 
reliably estimate the incremental effect of an additional month beyond this point. Therefore, we included a dichoto- 
mous variable to capture the average achievement level for this group. 
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with 16.6 points, or .34 of a standard deviation, on the TALS. In other words, the difference in 
TALS literacy associated with 18 months of adult education appears to be similar to the differ- 
ence in literacy scores associated with 18 months of formal preprogram education. In short, it 
appears that while there is no significant relationship between short-term participation in adult 
education and postprogram literacy achievement, there is a substantial association between liter- 
acy outcomes and longer-term participation in adult education. Table 3.5 further illustrates this 
point by presenting the predicted TALS scores (as well as other outcomes) for sample members 
with 0, 1, 6, 12, and 18 months of adult education. 

2. Math Achievement. Table 3.3 presents regression estimates of the relationship be- 
tween total months in adult education and postprogram Comprehensive Adult Student Assess- 
ment System (CASAS) math scores. These estimates were generated in the same manner as those 
presented in Table 3.2. However, as was the case when we compared Figures 3.1 and 3.2, the es- 
timates in this table suggest that the relationship between months in adult education and postpro- 
gram math scores differed from that which was observed for postprogram literacy scores. Our 
analysis suggested a relationship between participation in adult education and postprogram liter- 
acy that was weak during the early months and became stronger after the first year. The relation- 
ship between participation in adult education and postprogram math scores, on the other hand, 
suggests a relationship that was somewhat stronger during the early months of participation and 
weakened over time. 

The estimated relationship between total months of participation during the two-year fol- 
low-up and postprogram math scores, presented in column 1 of Table 3.3, suggests that, on average, 
an additional month of adult education completed during the two years following random assign- 
ment was associated with a difference of .28 points, or .020 of a standard deviation, in postprogram 
CASAS math scores. Thus, six months of adult education were associated with a gain of 1.68 
points, or .12 of a standard deviation, on the CASAS. As mentioned earlier, we estimated the rela- 
tionship between sample members’ accumulated education at program entry and their baseline test 
scores. According to these estimates, an additional year of education was associated with a differ- 
ence of 1.43 points, or .10 of a standard deviation, on the CASAS. Therefore, on average, the pay- 
off to six months of adult education in terms of math skills was as large as or larger than the payoff 
to an average year of the formal schooling received before entering the study. 

As in Table 3.2, the second column of Table 3.3 explores the possibility that the relation- 
ship between months of education and postprogram achievement was nonlinear. The estimates 
again reflect the simple pattern presented in graphic form in an earlier section. As in Figure 3.1, 
the estimates indicate the existence of a significant positive relationship between additional edu- 
cation and math achievement in the first six months, and no such relationship thereafter. In par- 
ticular, after the first month spent in adult education, each additional month through the sixth was 
associated with a difference of .78 points, or .056 of a standard deviation, on the CASAS. Ac- 
cording to this estimate, the difference in postprogram math achievement between someone who 
participated in adult education for one month and someone who participated for six months was 
3.9 points, or .28 of a standard deviation. 

After the first year of participation, there were no statistically significant gains in math 
scores associated with additional months of participation in adult education. In other words, our 
estimates predicted significant gains in math skills for those completing 6 months of adult educa- 
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National Evaluation of Welfare-to-Work Strategies 
Table 3.3 

For Sample Members Without a High School Diploma or GED at Random Assignment: 3 
Relationship Between Participation in Adult Education Activities and 
Two-Year CASAS Math Scores 



Coefficient 


Independent Variable 


Model 1 


Model 2 




Participation in adult education within 








2 years of random assignment 








1 month of participation 


-0.83 


0.42 






(2.64) 


(2.69) 




Total months, beyond 1 


0.28 *** 


- 






(0.10) 


- 




Months beyond 1, through 6 


- 


0.78 ** 






- 


(0.34) 




Months beyond 6, through 1 2 


- 


0.35 






- 


(0.43) 




Months beyond 12, through 18 


- 


0.49 






- 


(0.74) 




More than 18 months 


- 


-6.12 






- 


(4.23) 




Reading and math skills at random 








assignment 








TALS document literacy test score 


0.10 *** 


0.10 *** 






(0.01) 


(0.01) 




CASAS math test score 


0.28 *** 


0.28 *** 


' 




(0.03) 


(0.03) 




Education status at random assignment 








Highest grade completed 


0.54 * 


0.51 * 






(0.28) 


(0.28) 




Enrolled in education or training 


0.02 


-0.10 




activity in 12 months prior to 


(0.88) 


(0.88) 




random assignment 








Attitudes and opinions at random 








assignment 5 








Family or Personal Problems scale 


0.66 


0.62 






(0.72) 


(0.72) 




Parental Concerns scale 


0.81 


0.79 






(0.89) 


(0.89) 




Preference for School scale 


-2.25 *** 


-2.22 *** 






(0.72) 


(0.72) 




Depressive Symptoms scale 


0.39 


0.41 






(0.38) 


(0.38) 




Mastery scale 


-1.24 * 


-1.26 * 






(0.70) 


(0.70) 
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Table 3.3 (continued) 



Coefficient 


Independent Variable 


Model 1 


Model 2 


Family status at random assignment 






Ever married 


0.23 


0.27 




(0.84) 


(0.84) 


2 children 


0.52 


0.46 




(0.82) 


(0.82) 


3 or more children 


0.97 


0.79 




(0.92) 


(0.92) 


Any child 5 years old or younger 


0.00 


-0.14 




( 1 . 12 ) 


(1.13) 


Public assistance and labor force 






status at random assignment 






Cumulative years on public assistance 


-0.39 


-0.34 




(0.25) 


(0.25) 


First spell of AFDC receipt 


0.58 


0.87 




(1.18) 


(1.18) 


Any paid work in year prior to random 






assignment 


0.00 


0.00 




( 0 . 00 ) 


( 0 . 00 ) 


Demographic characteristics 






Female 


-1.07 


-1.14 




(1.61) 


(1.60) 


Age 


-0.13 ** 


-0.13 ** 




(0.06) 


(0.06) 


Black 


-3.51 *** 


-3 47 *** 




( 1 . 00 ) 


( 1 . 00 ) 


Not black or white 


-1.79 


- 1.86 




( 1 . 22 ) 


( 1 . 22 ) 


R~squared 


0.45 


0.45 


Sample size 


1,139 


1,139 
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Table 3.3 (continued) 

SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy 
test and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma 
or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus 
were determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * = 
10 percent; ** = 5 percent; and *** = 1 percent. 

Numbers shown in parentheses are the standard errors associated with the coefficient shown above. 

‘‘The calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult 
education within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who 
participated for 21 months or more were excluded due to small sample sizes. 

These subgroups are based on scales created from Private Opinion Survey data. For an explanation of these 
subgroups see Appendix E. 



tion, as opposed to none, but no significant differences in math scores for those completing 18 
months, as opposed to 12. Table 3.5 further illustrates this point by presenting the predicted CA- 
SAS scores for sample members with 0, 1, 6, 12, and 18 months of adult education. 

3. GEB Receipt. Table 3.4 presents estimates of the relationship between participa- 
tion in adult education and GED receipt during the follow up-period. Again, the differences be- 
tween the estimates in columns 1 and 2 of this table highlight the nonlinear nature of this rela- 
tionship. The first column reveals that, on average, an additional month of adult education was 
associated with an increase of .49 percentage points in the likelihood of receiving a GED or a 
high school diploma during the follow-up period. This suggests that a difference of six months in 
the time spent in adult education was associated with an average difference of 2.94 percentage 
points in the likelihood of obtaining a GED during the follow-up period. 

Column 2 reveals that this modest association between months spent in adult education 
and subsequent GED receipt represented a strong positive association during the first six months 
of participation, followed by a somewhat negative association after that. In particular, from the 
first through the sixth month, each additional month of adult education was associated with an 
increase of 4.18 percentage points in the average likelihood of obtaining a GED during the fol- 
low-up period. This means that those who participated for six months had a likelihood of receiv- 
ing a GED that was 21 percentage points higher than those who participated for only one month. 
On the other hand, each additional month of adult education from the seventh through the twelfth 
month of participation was associated with a reduction of 1.28 percentage points in this likeli- 
hood of receiving a GED during the follow-up period. A similar negative relationship was found 
for participation spells between 1 3 and 1 8 months in duration, but this relationships was not sta- 
tistically significant. 

Table 3.5 further illustrates this point (and summarizes earlier findings regarding reading 
and math scores) by presenting the estimated percentage of individuals who received their GED 
for sample members with 0, 1, 6, 12, and 18 months of adult education, respectively. Overall, 
this pattern suggests that there was a substantial positive association between months spent in 
adult education and receipt of a GED or high school diploma during the first six months of adult 
education. After that, this relationship weakened, becoming substantially negative over time. 
This suggests the possibility that individuals who stayed in adult education for longer periods 
may have been more difficult to serve, because of the presence of learning disabilities, their atti- 
tudes toward education, or other factors that made it more difficult for them to move toward the 
completion of a GED. On the other hand, individuals who received their GED after only a few 
months of adult education may have needed only a brief “refresher” course in order to pass the 
exam. As noted earlier, the interpretation of this apparent relationship is complicated by the fact 
that GED receipt may have caused spells of adult education to be shortened. This would dampen 
any positive association between participation in adult education and receipt of the credential and 
could explain why estimates describing this association became negative in the long run. 

C. The Effectiveness of Mandatory Admit Education 

Among other things, the NEWWS Evaluation was designed to evaluate the effectiveness 
of various mandatory welfare-to-work programs, including those that emphasize the receipt of 
additional education as a first step toward self-sufficiency. One of the primary questions with re- 




i i \ n 

-85- lUfc 



National Evaluation of Welfare-to-Work Strategies 
Table 3.4 



For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Relationship Between Participation in Adult Education Activities and 
GED Receipt at Two-Year Follow Up 





Coefficient 




Independent Variable 


Model 1 


Model 2 


Participation in adult education within 
2 years of random assignment 


1 month of participation 


-0.64 


6.43 




(4.04) 


(4.11) 


Total months, beyond 1 


0.49 *** 


- 




(0.15) 


- 


Months beyond 1 , through 6 


- 


4.18 *** 




- 


(0.55) 


Months beyond 6, through 12 


- 


-1.28 * 




- 


(0.65) 


Months beyond 12, through 18 


- 


-1.45 




- 


(1.14) 


More than 1 8 months 


- 


6.10 




- 


(6.44) 


Reading and math skills at random 
assignment 


TALS document literacy test score 


0.05 *** 


0.05 *** 




(0.02) 


(0.02) 


CASAS math test score 


0.28 *** 


0.28 *** 




(0.05) 


(0.05) 


Education status at random assignment 


Highest grade completed 


0.16 


0.17 




(0.46) 


(0.45) 


Enrolled in education or training 


2.83 ** 


2.63 * 


activity in 12 months prior to 


(1.40) 


(1.38) 


random assignment 

Attitudes and opinions at random 
assignment 6 


Family or Personal Problems scale 


-1.82 


-1.50 




(1.14) 


(1.13) 


Parental Concerns scale 


2.10 


1.76 




(1.32) 


(1.31) 


Preference for School scale 


0.50 


0.62 




(1.20) 


(1.18) 


Depressive Symptoms scale 


1.85 *** 


1.86 *** 




(0.61) 


(0.61) 


Mastery scale 


-2.00 * 


-2.21 ** 




(1.14) 


(1.13) 



(continued) 





Table 3.4 (continued) 



Coefficient 


Independent Variable 


Model 1 


Model 2 


Family status at random assignment 






Ever married 


-1.81 


-1.63 




(1.31) 


(1.29) 


2 children 


-0.90 


-0.64 




(1.34) 


(1.33) 


3 or more children 


-0.49 


-0.49 




(1.48) 


(1.46) 


Any child 5 years old or younger 


-0.14 


-0.16 




(1.33) 


(1.31) 


Public assistance and labor force 






status at random assignment 






Cumulative years on public assistance 


-0.49 


-0.36 




(0-42) 


(0.42) 


First spell of AFDC receipt 


5 45 *** 


5.87 *** 




(1.74) 


(1.72) 


Any paid work in year prior to 


0.00 


0.00 


random assignment 


( 0 . 00 ) 


( 0 . 00 ) 


Demographic characteristics 






Female 


2.39 


2.50 




(3.30) 


(3.26) 


Age 


-0.14 


- 0.10 




( 0 . 10 ) 


( 0 . 10 ) 


Black 


-1.28 


-1.41 




(1.59) 


(1.57) 


Not black or white 


-1.57 


-1.38 




(1.89) 


( 1 . 86 ) 


R-squared 


0.24 


0.30 


Sample size 


1,962 


1,962 
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Table 3,4 (contsoued) 



SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test 
and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS - Test of Applied Literacy Skills. 

CASAS - Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 3 1, counting the month in which random 
assignment occurred as month 1. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * = 10 
percent; — 5 percent; and *** = 1 percent. 

Numbers shown in parentheses are the standard errors associated with the coefficient shown above. 

‘‘The calculations in this table include only sample members who had both baseline and follow-up test score data 
in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult education within 
2 years of random assignment for more than 1 month but less than 21 months. Individuals who participated for 21 
months or more were excluded due to small sample sizes. 

b These subgroups are based on scales created from Private Opinion Survey data. For an explanation of these 
subgroups see Appendix E. 





- 88 - 



National Evaluation of Welfare-to-Work Strategies 
Table 3.5 



For Sample Members Without a High School Diploma or GED at Random Assignment: 3 
Predicted Postprogranu Test Scores and GED Receipt, 
by Months of Participation in Adult Education 



Months of Participation 


Outcome 


0 Months 


1 Month 


6 Months 


12 Months 


18 Months 


GED receipt 


1.64 


8.07 


28.97 


21.29 


12.59 


Difference 




6.43 


20.90 


- 7.67 


-8.70 


Effect size b 




0.27 


0.87 


-0.32 


- 0.36 


Sample size 




1,962 








TALS document literacy test score 


247.23 


251.15 


256.65 


250.91 


270.71 


Difference 




3.91 


5.50 


- 5.74 


19.80 


Effect size b 




0.08 


0.11 


-0.12 


0.40 


Sample size 




1,955 








CASAS math test score 


206.76 


207.18 


211.10 


213.17 


216.11 


Difference 




0.42 


3.92 


2.07 


2.94 


Effect size b 




0.03 


0.28 


0.15 


0.21 


Sample size 




1,139 









SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy test 
and CASAS math test data, and the JOBS Two-Year Client Survey. 



NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were 
determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1 . 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * = 

10 percent; ** = 5 percent; and *** = 1 percent. 

^he calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult education 
within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who participated 
for 21 months or more were excluded due to small sample sizes. 

^he effect size equals the difference divided by the standard deviation of the outcome measure for the control 
group. The standard deviations for control group members are: 24 for GED receipt; 49 for the TALS document 
literacy test score; and 14 for the CASAS math test score. 
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spect to these programs is whether the benefits of mandatory participation in adult education are 
the same as the benefits of voluntary participation. If, for example, welfare recipients who par- 
ticipated in adult education as part of a mandatory program were significantly less likely to bene- 
fit from this participation than those whose participation was entirely voluntary, the implication 
would be that welfare agencies are limited in their ability to better prepare individuals for the la- 
bor market by requiring them to improve their education. 

As was mentioned earlier in this chapter, at the outset of the evaluation, individuals in the 
three-site sample were randomly assigned either to a mandatory education-focused welfare-to- 
work program or to a control group whose members could access available education and train- 
ing services on their own. Even though they were not required to do so, some members of the 
control group reported participating in adult education activities during the two-year follow-up 
period. The following analysis compares the payoff to participation in adult education for mem- 
bers of both the program and the control groups to ascertain whether and how this payoff varied 
depending on whether or not participation was mandatory. 

Specifically, the analyses of the relationship between additional months of adult educa- 
tion and the educational outcomes discussed above were repeated, allowing the payoff to adult 
education activities to vary according to whether or not individuals were part of the program 
group or the control group. Based on these regression estimates, Tables 3.6 and 3.7 present pre- 
dicted levels of reading scores and GED receipt for individuals who spent different amounts of 
time in adult education, by research group. Note that because the postprogram CASAS test scores 
were administered only to a subgroup of the sample, the sample size was not large enough to al- 
low us to conduct subgroup analyses for this outcome. 13 

1. Reading Skills. The first panel of Table 3.6 shows predicted postprogram TALS 
scores among the individuals in mandatory education-focused programs and among individuals 
who were in the control group, at 1, 6, 12, and 18 months of participation in adult education. The 
table also reports the results of statistical tests evaluating the hypothesis that the relationships 
shown were the same across these two groups. It appears that the payoff to adult education was 
similar for both groups, with effects somewhat stronger for control group members in the early 
months and for program group members in the later months. While the difference was not statis- 
tically significant, this pattern suggests that the program mandate may have improved the payoff 
to persistence in adult education among those who needed more than a year of adult education to 
achieve real gains in reading skills. 

2. GED Receipt. While the relationship between postprogram test scores and months 
of adult education was not significantly affected by whether or not participation occurred under a 
mandate, the payoff in terms of GED receipt was. The second panel of Table 3.7 reports the pre- 
dicted percentage of students receiving a GED during the follow-up period — among students 
with 1, 6, 12, and 18 total months of adult education — separately for individuals who were and 



,3 In particular, because welfare recipients with young children were administered a lengthy survey module re- 
garding child outcomes, they were not administered the CASAS math test at the time of the follow-up interview. 
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National Evaluation of Welfare-to-Work Strategies 
Table 3.6 



For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Predicted TALS Document Literacy Test Score, 
by Subgroup and Months of Participation in Adult Education 







Months of Participation 




Subgroup 


1 Month 


6 Months 


12 Months 


18 Months 


Full Sample 


251.15 


256.65 


250.91 


270.71 


Difference 




5.50 


- 5.74 


19.80 


Effect size b 




0.11 


-0.12 


0.40 


Mandate 


HCD Group 


253.36 


255.61 


255.92 


274.061 


Difference 




2.25 


0.31 


18.14 


Effect size b 




0.05 


0.01 


0.37 


Control Group 


245.41 


257.43 


232.21 


263.42 


Difference 




12.03 


-25.22 


31.20 


Effect size b 




0.25 


-0.51 


0.64| 


Site 


Atlanta 


249.86 


260.89 


261.75 


274.281 


Difference 




11.04 


0.86 


12.53 


Effect size b 




0.23 


0.02 


0.26 


Grand Rapids 


262.16 


275.73 


265.54 


285.87 


Difference 




13.57 


-10.19 


20.33 


Effect size b 




0.28 


-0.21 


0.41 


Riverside 


234.90 


224.36 


229.19 


265.98 


Difference 




-10.54 


4.82 


36.79 1 


Effect size b 




-0.22 


0.10 


O.75] 


Hours per Week 


Less than 20 hours per week 


254.29 


260.02 


250.45 


276.231 


Difference 




5.73 


- 9.57 


25.79 


Effect size b 




0.12 


-0.20 


0.53 


20 or more hours per week 


240.77 


241.38 


251.94 


251.17 


Difference 




0.61 


10.56 


- 0.76 


Effect size b 




0.01 


0.22 


-0.02] 



(continued) 




- 9 >-103 



Table 3.6 (continued) 







Months of Participation 




Subgroup 


1 Month 


6 Months 


12 Months 


18 Months 


Initial Academic Achievement 










"Normal" academic achievement 


252.16 


257.33 


252.40 


274.79] 


Difference 




5.17 


-4.93 


22.39 


Effect size b 




0.11 


-0.10 


0.46 


High initial CASAS math test score 


209.56 


214.41 


208.17 


226.61 


Difference 




4.85 


-6.24 


18.43 


Effect size b 




0.10 


-0.13 


0.38 


High initial TALS literacy score 


279.70 


285.88 


280.79 


299.76 


Difference 




6.17 


-5.08 


18.97 


Effect size b 




0.13 


-0.10 


0.39J 


Highest Grade Completed 










Grade 8 or below 


233.05 


231.04 


256.60 


261.43] 


Difference 




-2.01 


25.56 


4.83 


Effect size b 




-0.04 


0.52 


0.10 


Grade 9-10 


252.37 


262.88 


246.77 


272.26 


Difference 




10.51 


-16.11 


25.48 


Effect size b 




0.21 


-0.33 


0.52 


Grade 1 1 


253.44 


253.64 


254.02 


268.38 


Difference 




0.20 


0.38 


14.35 


Effect size b 




0.00 


0.01 


0.29j 


Sample size 


1,955 








(continued) 
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Table 3.6 (continued) 

SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy 
test and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, 
and thus were determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

HCD = Human Capital Development. 

CASAS = Comprehensive Adult Student Assessment System. 

TALS = Test of Applied Literacy Skills. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1 . 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * 

= 10 percent; ** = 5 percent; and *** = 1 percent. 

Brackets around groups of coefficients within models indicate that an F-test of joint significance was applied 
to these variables. This statistic tests the hypothesis that the bracketed coefficients are statistically distinguishable 
from zero. Statistical significance levels are indicated as: t = 10 percent; tt = 5 percent; and ftt = 1 P ercent - 

^he calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education- focused programs and had participated in adult 
education within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who 
participated for 21 months or more were excluded due to small sample sizes. 

b The effect size equals the difference divided by the standard deviation of the outcome measure for the 
control group. The standard deviation for control group members of the TALS document literacy test score is 49. 
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National Evaluation of Welfare-to-Work Strategies 
Table 3.7 

For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Predicted Percentage of Students Receiving a GED, 
by Subgroup and Months of Participation in Adult Education 



Months of Participation 



Subgroup 


1 Month 


6 Months 


12 Months 


18 Months 


Full Sample 


8.07 


28.97 


21.29 


12.59 


Difference 




20.90 


- 7.67 


-8.70 


Effect size b 




0.87 


-0.32 


-0.36 


Mandate 










HCD Group 


10.27 


25.63 


23.91 


ttt 

25.001 


Difference 




15.35 


-1.72 


1.10 


Effect size b 




0.64 


-0.07 


0.05 


Control Group 


2.27 


29.81 


0.00 


0.00 


Difference 




27.54 


-29.81 


0.00 


Effect size b 




1.15 


-1.24 


o.ooj 


Site 


















ttt 


Atlanta 


15.16 


49.78 


45.62 


50.82] 


Difference 




34.62 


-4.17 


5.20 


Effect size b 




1.44 


-0.17 


0.22 


Grand Rapids 


17.31 


31.47 


31.66 


-3.53 


Difference 




14.16 


0.19 


-35.18 


Effect size b 




0.59 


0.01 


-1.47 


Riverside 


0.00 


4.44 


0.00 


73.62 


Difference 




4.44 


-4.44 


73.62 


Effect size b 




0.19 


-0.19 


3.07] 


Hours per Week 










Less than 20 hours per week 


12.57 


21.45 


18.26 


23.47] 


Difference 




8.89 


-3.20 


5.22 


Effect size b 




0.37 


-0.13 


0.22 


20 or more hours per week 


0.00 


30.77 


22.66 


32.36 


Difference 




30.77 


-8.11 


9.70 


Effect size b 




1.28 


-0.34 


0.4q] 



(continued) 
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Table 3.7 (continued) 



Months of Participation 



Subgroup 


1 Month 


6 Months 


12 Months 


18 Months 


Initial Academic Achievement 


















ttt 


"Normal" academic achievement 


1.25 


2.05 


6.24 


2.931 


Difference 




0.79 


4.19 


-3.31 


Effect size b 




0.03 


0.17 


-0.14 


High initial CASAS math test score 


20.00 


52.11 


39.33 


36.87 


Difference 




32.11 


- 12.78 


-2.46 


Effect size b 




1.34 


-0.53 


-0.10 


High initial TALS literacy score 


1.54 


14.44 


9.91 


16.06 


Difference 




12.90 


-4.53 


6.16 


Effect size b 




0.54 


-0.19 


0.26] 


Highest Grade Completed 








ft 


Grade 8 or below 


0.00 


1.12 


0.00 


0.24] 


Difference 




1.12 


-1.12 


0.24 


Effect size b 




0.05 


-0.05 


0.01 


Grade 9-10 


12.87 


23.60 


16.63 


27.83 


Difference 




10.73 


- 6.97 


11.19 


Effect size b 




0.45 


-0.29 


0.47 


Grade 11 


10.20 


28.87 


36.49 


12.37 


Difference 




18.67 


7.62 


-24.12 

I 


Effect size b 




0.78 


0.32 


-l.OlJ 


Sample size 


1,962 
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Table 3.7 (continued) 

SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy 
test and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, 
and thus were determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

HCD = Human Capital Development. 

CASAS = Comprehensive Adult Student Assessment System. 

TALS = Test of Applied Literacy Skills. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1. 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * 

= 10 percent; ** = 5 percent; and *** = 1 percent. 

Brackets around groups of coefficients within models indicate that an F-test of joint significance was applied 
to these variables. This statistic tests the hypothesis that the bracketed coefficients are statistically distinguishable 
from zero. Statistical significance levels are indicated as: f = 10 percent; ff = 5 percent; and fft = 1 percent. 

^he calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult 
education within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who 
participated for 21 months or more were excluded due to small sample sizes. 

b The effect size equals the difference divided by the standard deviation of the outcome measure for the 
control group. The standard deviation for control group members of GED receipt is 24. 



were not under a mandate to participate. 14 The statistical test reported in the table indicates that 
there was a statistically significant difference in the estimated payoff for the two research groups. 

Furthermore, the estimates reported in the table suggest that the relationship between 
GED receipt and months spent in adult education during the follow-up period was somewhat 
stronger among individuals whose participation was voluntary. In particular, among those who 
were under a mandate, 10 percent of individuals with only one month of education received their 
GED by the end of the follow-up period, while the same can be said for only 2 percent of their 
counterparts in the control group. However, among welfare recipients who reported six months 
of education, 26 percent of the group that was under the mandate reported obtaining a GED dur- 
ing the follow-up period, compared with 30 percent of the group that was not. In other words, 
while the percentage of voluntary adult education participants receiving a GED during the first 
month of adult education lagged behind that of the group under the mandate, by six months into 
their participation in adult education this pattern was reversed. It is possible that those who chose 
to participate in adult education to get a GED had a greater need for additional education than 
those who were placed in adult education by a mandatory welfare-to-work program. It is likely 
that control group members who were close to passing the GED test just took it without formal 
preparation, while those in the program group were first enrolled in formal adult education pro- 
grams. 

D. Site-Specific Differences in the Payoff to Additional Admit Edmcation 

Another important question to address is whether there were systematic differences in the 
payoff to adult education across the three programs we studied. If such differences existed, they 
would warrant further exploration of variation in sample characteristics and program approaches 
across the sites. 

The third panel of Table 3.6 shows predicted postprogram TALS document literacy test 
scores across individuals who reported receiving different amounts of adult education, sepa- 
rately for Atlanta, Grand Rapids, and Riverside. The observed patterns suggest that there were 
some differences in the payoff to adult education across these sites. First of all, sample mem- 
bers’ test scores at the outset of their participation in adult education were substantially lower 
in Riverside than in the other sites. In particular, the predicted postprogram TALS scores 
among individuals in the Riverside site who reported only one month of adult education were 
approximately 15 points, or .31 of a standard deviation, lower than the scores of their counter- 
parts in Atlanta, and about 27 points, or .55 of a standard deviation, lower than the scores of 
their counterparts in Grand Rapids. Moreover, in Riverside, the overall differences in postpro- 
gram test scores across individuals with 12 as opposed to 18 months of participation appear to 
be smaller than in the other two sites. However, the variation in patterns across the three sites 
was not statistically significant. 



14 The predicted values in this table were based on a ordinary least squares regression. Predicted percentages 
were truncated to remain between zero and 100. Thus, the predicted values of zero that are observed in the table ac- 
tually represent negative predicted values. While a logit or probit estimate would have accomplished this automati- 
cally, the data in this sample were not sufficient to generate these maximum likelihood estimates. Moreover, the 
nonlinear specification of the key independent variable reduces the likelihood of any bias in the estimates caused by 
restrictions of linearity. 
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The third panel of Table 3.7 reports site- specific estimates of the relationship between 
participation in adult education and GED receipt. The table suggests that there was statistically 
significant variation in this relationship across the three sites. As with postprogram literacy 
scores, the relationship between participation in adult education and GED receipt was stronger in 
Grand Rapids and Atlanta but weaker in Riverside. While this may indicate that education pro- 
grams were stronger in Grand Rapids and Atlanta, it also could reflect differences in rules and 
policies used by the welfare-to-work programs. 

E. Individual Characteristics and the Payoff to Time Spent in Adult Education 

In addition to evaluating the effectiveness of adult education programs in general, an im- 
portant goal of the NEWWS Evaluation is to assess to what extent the effectiveness of adult edu- 
cation programs differs across various subgroups. Tables 3.6 and 3.7 present predicted outcome 
levels for individuals who reported different amounts of adult education, within several sub- 
groups defined using individual characteristics and program experiences. These estimates are 
based on regressions which allowed the payoff to adult education to vary across subgroups. 

1. Hours per Week. Throughout this chapter, we have examined the relationship be- 
tween academic achievement and the length of students’ stay in adult education, without regard 
for the reported number of hours spent in class each week. At the end of the follow-up period, 
students who participated in adult education were asked to report the average number of hours 
dedicated to that activity each week. To the extent that the hours per week reflect the intensity of 
the educational experience or the degree of students’ engagement, we might expect those who 
participated for more hours each week to experience a stronger payoff to the time they spent in 
adult education. In order to explore this question Tables 3.6 and 3.7 report predicted TALS and 
GED outcomes separately for those who reported spending 20 or more hours per week in adult 
education and those who reported attending less than 20 hours per week. 15 

With respect to postprogram TALS scores, the statistical test reported in Table 3.6 indi- 
cates that there was no statistically significant difference in the payoff estimated for these two 
groups. However, predicted outcome levels for individuals who reported receiving different 
amounts of adult education during the follow-up period do suggest some interesting patterns. In 
particular, students who attended classes for 20 or more hours per week began with lower scores 
and experienced earlier gains, which ceased to accrue after 12 months in the program. On the 
other hand, individuals who attended for less than 20 hours per week appeared to start with 
higher test scores and experienced relatively small benefits to their participation until after a year 
in the program. 

With respect to GED receipt, the predicted outcome levels indicate that the positive re- 
lationship between length of stay in adult education and GED receipt is stronger for students 
who dedicate more hours to this activity each week. As might be expected, sample members 
who participated for less than 20 hours per week had a higher initial likelihood of receiving the 



l5 Note that hours per week were measured retrospectively, at the two-year follow-up interview. Often, at least 
several months had passed between the last spell of participation in adult education and the interview. Therefore, it is 
possible that this variable was measured with considerable error. If so, it is possible that measurement error has bi- 
ased these coefficient estimates toward zero. 
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credential but experienced a smaller payoff to additional participation. These differences were 
statistically significant. 

2. Initial Academic Preparation. Among the important questions to consider is the 
extent to which benefits to participation in adult education vary with sample members’ level of 
academic preparation. To the extent that benefits are concentrated among those with different 
levels of academic achievement or vary across students with different levels of formal education, 
programs may decide to target their adult education resources differently or adjust their expecta- 
tions and time lines. To answer this question, we estimated several regression models which al- 
lowed the payoff to adult education to differ depending on individuals’ initial education and aca- 
demic achievement. The results suggest that initial academic preparation did not affect the rela- 
tionship between months of adult education and postprogram literacy scores. However, as shown 
in Table 3.7, those with higher initial test scores were more likely to benefit from subsequent par- 
ticipation in adult education by obtaining a GED credential. 

The final panel of Table 3.7 breaks down the relationship between participation in adult 
education and GED receipt for sample members with different levels of formal education. The 
results suggest a strong relationship between the payoff to adult education and the highest grade 
completed before entering adult education. In particular, individuals are divided into three 
groups, those who completed 8th grade or lower, those who completed 9th or 10th grade, and 
those who completed 11th grade. It appears that the relationship between adult education and 
GED receipt is stronger for those with who entered the study with more formal education. In par- 
ticular, the predicted percentages of GED receipt among individuals with eight years or less edu- 
cation suggest that virtually none of them received their GED during the follow-up period and 
that the payoff to additional months of education in terms of GED receipt is extremely limited. 6 
This is consistent with the finding from Chapter 2 that the impact of assignment to an education- 
focused program had a limited effect on GED receipt for individuals who started with an 8th 
grade education or less. Understandably, it appears to be difficult for an adult education program 
to make up for so many years of formal education, especially within a two-year follow-up period. 

On the other hand, individuals who had completed 9 years or more of education at ran- 
dom assignment experienced a substantial payoff to participation in adult education in terms of 
their likelihood of receiving a GED. For example, among those who completed 9th or 10th grade 
before random assignment, 13 percent of individuals completing only 1 month of adult education 
received their GED during the follow-up period, while 24 percent of the individuals completing 6 
months of adult education did the same. This difference was even larger for sample members 
who completed 1 1 years of formal education before participation in adult education. Moreover, 
while individuals with 9 or 10 years of formal education experienced a decline in GED receipt 
between 6 and 12 months of adult education, individuals who completed 11th grade experienced 
an increase from 29 to 36 percent over this same period of involvement. 



l6 Note that these predicted percentages are generated from a ordinary least squares regression, with the pre- 
dicted values restricted to fall between zero and 100. To the extent that the actual predicted values were negative, 
this may slightly underestimate the payoff to additional months of education among this subgroup of students. 
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3. Barriers to Participation!. Earlier evidence suggests the possibility that barriers to 
participation in adult education, such as having young children or having family or personal 
problems, might negatively affect participants’ ability to benefit from their participation. For ex- 
ample, evidence presented in Chapter 2 indicates that random assignment to an education- 
focused welfare-to-work program had the smallest effect on participation in adult education 
among clients who reported having a high level of family or personal problems. In order to ex- 
plore whether or not these factors limited the payoff to the time individuals spent in adult educa- 
tion, the relationship between months of education and GED receipt as well as postprogram 
TALS document literacy test scores were estimated separately for individuals with many family 
or personal problems and parental concerns and for individuals who had children younger than 6 
in their household. The results indicate that there was no significant relationship between these 
barriers and the relationship between educational outcomes and additional months of adult 
education. Chapter 2 found that greater barriers to participation were associated with less 
favorable program effects on educational outcomes. Our findings in this analysis suggest that 
lower rates of participation might explain these differences, rather than smaller payoffs for those 
who did participate. 

4. Attitudes Toward School. Another important issue to address is whether the rela- 
tionship between time spent in adult education activities and postprogram test scores depends on 
students’ attitudes toward school. To explore this hypothesis, we estimated the relationship be- 
tween months of participation and postprogram educational outcomes, allowing it to vary accord- 
ing to students’ disposition toward school. This was measured by whether students reported any 
participation in adult education activities over the past year and whether the students reported 
having a preference for school. If a student reported either of these, then that student was consid- 
ered to have a positive attitude toward adult education. The estimates suggest that student atti- 
tudes toward adult education did not affect the payoff. Together with the findings reported in 
Chapter 2, this suggests that welfare-to-work programs can successfully overcome participants’ 
initial distaste for adult education. Such lack of motivation did not affect program impacts on 
participation rates, and it apparently did not affect the payoff to participation either. 

F. Provider Characteristics and the Payoff to Time Spent in Adult Education 

The previous section examined whether the relationship between participation and educa- 
tion outcomes varied with participants’ individual characteristics. This section explores to what 
extent the payoff from adult education varied across the schools and agencies that provided adult 
education services to sample members in our study. 17 Did certain types of providers improve 
educational outcomes more than others? Did teachers’ experience and education matter? 

1. Teachers 5 Experience. To explore the effects of teachers’ experience on the payoff 
to adult education, we estimated the usual relationship between participation in adult education 
and educational outcomes, allowing it to vary with the average years of teaching experience of 



l7 Not all the individuals in the sample attended classes at providers who were surveyed as part of the teacher 
survey. Therefore, the sample of individuals upon which we base our estimates of the effect of provider characteris- 
tics on the payoff to adult education is different from the sample of individuals upon which we have based our esti- 
mates of the payoff to education thus far. 
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teachers at each adult education provider. 18 Tables 3.8 and 3.9 present predicted postprogram 
outcomes for individuals who received different amounts of adult education, estimated separately 
for different levels of experience among the teachers at each provider. In particular, we predicted 
educational outcomes for sample members who attended classes at hypothetical providers whose 
teachers had 15 as opposed to 25 years of experience. While these levels of experiences may 
seem somewhat high, it should be noted that the adult education providers surveyed employed 
relatively experienced teachers. In particular, the average level of teacher experience was ap- 
proximately 17 years, while the standard deviation of teacher experience was about 7. 

With respect to postprogram test scores, the statistical test reported in Table 3.8 indicates 
that the payoff to months of education was significantly greater in classes with more experienced 
teachers. For example, when the average level of experience was 25 years, the difference in test 
scores between respondents with one month and six months of adult education was predicted to be 
9.07 points, or .19 of a standard deviation. When the average experience was 15 years, this payoff 
to five additional months of education dropped to 2.72 points, or .06 of a standard deviation. 

In terms of GED receipt, on the other hand, the payoff to adult education seems to have 
been negatively related to teachers’ experience. As shown in Table 3.9, at providers where the 
average teacher had 15 years of experience, the predicted difference in GED receipt between in- 
dividuals with one and six months of adult education was 1 1 percentage points. However, addi- 
tional months of education at providers with more experienced teachers (25 years on average) 
had no such positive effect. Although this difference is not statistically significant, the explana- 
tion for it is not obvious. One potential explanation is that teachers with different levels of ex- 
perience have different teaching approaches, and that teaching approaches which promote suc- 
cess in obtaining a GED may be different from approaches which promote increases in measured 
literacy. It does not, however, suggest that students at providers who employ more experienced 
teachers experience universally strong payoffs to adult education. 

2. Teachers’ Education. Tables 3.8 and 3.9 also report predicted postprogram TALS 
scores and GED receipt among individuals with varying levels of adult education, separated ac- 
cording to the percentage of teachers at their providers who possessed a master’s degree. The sta- 
tistical test reported in Table 3.9 suggests that there was no significant relationship between the 
payoff to adult education in terms of GED receipt and the proportion of teachers who had a mas- 
ter’s degree. While the predicted percentages in the table suggest that there might have been 



18 The actual specification interacted average experience and average experience squared, so that the possibility 
of diminishing returns could be accounted for. Also, remember that all the estimates of the relationship between time 
spent in adult education and postprogram education outcomes include a series of dummy variables, one for each pro- 
vider, in order to account for any unobserved provider-level variables that might have a direct effect on postprogram 
educational outcomes and might be correlated with time spent in adult education. Estimating these fixed effects 
avoids potential bias of the standard errors associated with the estimated effect of time spent in adult education. Un- 
fortunately, including the fixed effects precludes us from estimating the direct effects of provider characteristics on 
student outcomes; instead, we only can estimate the relationship between provider characteristics and the payoff to 
time spent in adult education. 

l9 More precisely, the average level of teacher experience across individuals in our sample — in other words, the 
average level of experience encountered by the average student at the provider — was 16.67 years, and the standard 
deviation was 6.67. Note also that the average years of experience includes all types of teaching experience, and does 
not necessarily equate to years of teaching at the adult level. 
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Table 3.8 



For Sample Members Without a High School Biploma or GEB at Random Assignment: 8 
Predicted TALS Bocuinent Literacy Test Score, 
by Provider Characteristics and Months of Participation 



Characteristic 




Months of Participation 




1 Month 


6 Months 


12 Months 


1 8 Months 


Average Teaching Experience 








X 


1 5 years 


233.63 


236.34 


253.46 


223.53] 


Difference 




2.72 


17.12 


-29.94 


Effect size b 




0.06 


0.35 


-0.61 


25 years 


233.63 


242.70 


237.59 


260.27 


Difference 




9.07 


-5.11 


22.69 


Effect size b 




0.19 


-0.10 


0.46] 


Teacher’s Education 










25% with master’s degree 


236.92 


236.96 


245.85 


239.42] 


Difference 




0.04 


8.90 


-6.43 


Effect size b 




0.00 


0.18 


-0.13 


50% with master’s degree 


236.92 


236.11 


239.16 


248.30 


Difference 




-0.80 


3.04 


9.15 


Effect size b 




-0.02 


0.06 


0.19 


75% with master’s degree 


236.92 


235.27 


232.47 


257.19 


Difference 




-1.65 


-2.81 


24.72 


Effect size b 




-0.03 


-0.06 


0.50j 


Sample size 


452 
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Table 3.8 (continued) 

SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy 
test and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, 
and thus were determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which 
random assignment occurred as month 1 . 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * 

= 10 percent; ** = 5 percent; and *** = 1 percent. 

Brackets around groups of coefficients within models indicate that an F-test of joint significance was 
applied to these variables. This statistic tests the hypothesis that the bracketed coefficients are statistically 
distinguishable from zero. Statistical significance levels are indicated as: f = 10 percent; tf = 5 percent; and ttt 
= 1 percent. 

“The calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult 
education within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who 
participated for 21 months or more were excluded due to small sample sizes. 

b The effect size equals the difference divided by the standard deviation of the outcome measure for the 
control group. The standard deviation for control group members of the TALS document literacy test score is 49. 
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National Evaluation of Welfare-to-Work Strategies 
Table 3.9 



For Sample Members Without a High School Diploma or GED at Random Assignment: 8 
Predicted Percentage of Students Receiving a GED, 
by Provider Characteristics and Months of Participation 



Characteristic 




Months of Participation 




1 Month 


6 Months 


12 Months 


18 Months 


Average Teaching Experience 


15 years 


23.85 


34.60 


36.02 


12.48] 


Difference 




10.76 


1.41 


-23.53 


Effect size b 




0.45 


0.06 


-0.98 


25 years 


23.85 


19.01 


22.09 


9.36 


Difference 




-4.84 


3.08 


-12.73 


Effect size b 




-0.20 


0.13 


-0.53J 


Teacher's Education 


25% with master's degree 


19.81 


24.72 


23.44 


19.651 


Difference 




4.91 


-1.27 


-3.80 


Effect size b 




0.20 


-0.05 


-0.16 


50% with master's degree 


19.81 


27.74 


22.93 


37.68 


Difference 




7.93 


-4.81 


14.75 


Effect size b 




0.33 


-0.20 


0.61 


75% with master's degree 


19.81 


30.76 


22.42 


55.72 


Difference 




10.95 


-8.34 


33.30 


Effect size b 




0.46 


-0.35 


1.39J 


Sample size 


360 
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Table 3.9 (contimned) 

SOURCES: MDRC calculations from information routinely collected by welfare staff, TALS document literacy 
test and CASAS math test data, Private Opinion Survey data, and the JOBS Two-Year Client Survey. 

NOTES: About a quarter of the sample in Riverside’s education- focused program did have a high school 
diploma or GED, but they scored low on the math or reading portion of the appraisal test or had limited English, 
and thus were determined to be in need of basic education. 

Adult education includes ABE, GED, ESL, and high school completion classes. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

Survey respondents were interviewed between month 25 and month 31, counting the month in which random 
assignment occurred as month 1 . 

To compensate for differences in the proportion of subgroup members chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted 
equally. 

A two-tailed t-test was applied to coefficients listed above. Statistical significance levels are indicated as: * 

= 10 percent; ** = 5 percent; and *** = 1 percent. 

Brackets around groups of coefficients within models indicate that an F-test of joint significance was applied 
to these variables. This statistic tests the hypothesis that the bracketed coefficients are statistically distinguishable 
from zero. Statistical significance levels are indicated as: f = 10 percent; ft = 5 percent; and ttt “ 1 percent. 

‘"The calculations in this table include only sample members who had both baseline and follow-up test score 
data in the Atlanta, Grand Rapids, and Riverside education-focused programs and had participated in adult 
education within 2 years of random assignment for more than 1 month but less than 21 months. Individuals who 
participated for 21 months or more were excluded due to small sample sizes. 

b The effect size equals the difference divided by the standard deviation of the outcome measure for the 
control group. The standard deviation for control group members of GED receipt is 24. 
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slight differences in the payoff to the first six months of adult education, these differences are not 
statistically significant. 

On the other hand, the evidence in Table 3.8 suggests that there was a relationship be- 
tween the percentage of the teachers who possessed advanced degrees and the payoff to time 
spent in adult education in terms of literacy skills. The predicted outcome levels as well as the 
statistical test reported in this table indicate that the relationship between TALS scores and 
months of adult education was significantly enhanced by the percentage of the teachers who had 
a master’s degree or higher. In the table we compare effects of additional months of education at 
two different hypothetical providers. Of the teaching staff at the first provider, it was assumed 
that 75 percent had advanced degrees, compared with 25 percent at the other provider. We then 
applied these numbers to the regression estimates and calculated the incremental gain in reading 
skills between 12 and 18 months of participation in adult education. We found that reading 
scores were not increased at the hypothetical provider with the less educated teaching staff 
(scores actually went down a little) and that scores increased by 25 points, or .50 of a standard 
deviation, at the other hypothetical provider. 20 

3. Other Provider Characteristics. Several other provider characteristics might be 
expected to influence the relationship between time spent in adult education and postprogram 
educational outcomes. To explore these, the JOBS teacher survey collected data on such factors 
as average class size, the degree of individual attention given to students, the proportion of staff 
who worked full time, the connection between the education provider and the JOBS program, the 
proportion of JOBS students in the classes at each provider, the instructional focus of the teach- 
ers, and the exit standards in place at each program. The distribution of these characteristics 
across providers was described and discussed in Chapter 2. Interestingly, teaching experience and 
education were the only characteristics that were significantly related to the payoff to time spent 
in adult education. 

Taken at face value, the findings regarding provider characteristics suggest that teachers’ 
qualifications are the primary provider-level factors which affect the payoff to adult education. In 
terms of literacy skills, students appear to benefit more when their time in adult education is 
spent with teachers who are both more experienced and more educated. Beyond this, other pro- 
vider characteristics do not seem to be important. 

It is possible, however, that this finding reflects a kind of “aggregation bias,” or “measure- 
ment error.” Because the provider characteristics used in these estimates were the average responses 
from the teachers who were interviewed at each provider, they thus may not have reflected stu- 
dents’ actual experiences as accurately as other data might have. For example, if we had collected 
data regarding students’ specific experiences, we might know the actual size of the class each stu- 
dent attended. Instead, we surveyed teachers, which only enabled us to measure a sense of the aver- 
age class size at each provider. Therefore, we can say that class size was measured loosely, or with 
some error. If this sort of measurement error was present, the estimated relationship would likely be 



20 Among the individuals in this sample, the mean proportion of teachers at their providers with an advanced de- 
gree is 52 percent, and the standard deviation is 25 percentage points. 



biased toward zero. 21 Therefore, to the extent that this phenomenon was present, we may have un- 
derestimated the effects of some provider characteristics on student outcomes. 

VI. Conclusions 

Overall, the evidence in this chapter suggests that, controlling for initial academic prepa- 
ration, the amount of time welfare recipients spent in adult education programs was significantly 
and positively related to postprogram math achievement, literacy achievement, and GED receipt. 
Apparently, the magnitude of the effect depended on the outcome in question as well as on the 
number of months of adult education an individual had already accumulated. For example, 
whereas during the first 12 months additional months of participation in adult education were 
almost completely unrelated to postprogram reading scores, the predicted difference between the 
postprogram TALS scores of individuals with 12 as opposed to 18 months of participation was 
about 20 points, or .40 of a standard deviation. This suggests that long-term participation in adult 
education programs, as currently structured, is needed to materially improve the reading skills of 
participants in welfare-to-work programs. 

For math scores, on the other hand, the payoff to additional months of adult education 
appears to have been concentrated during the first six months of participation. Specifically, the 
predicted difference between one and six months of participation was about 3.9 points, or .28 of 
a standard deviation, with little further gain associated with additional participation beyond that 
point. Therefore, to the extent that policymakers are interested in improving welfare recipients’ 
math skills, they might expect improvement to occur soon after the recipients begin participating 
in adult education. However, inasmuch as this effect appears to flatten out after six to eight 
months, the ability to improve math skills further by these programs as currently structured ap- 
pears limited. We cannot be sure that these relationships are causal. However, the delay in the 
effect of reading scores — combined with the fact that reading and math scores do not follow a 
similar pattern — undermines the theory that individuals with more months in adult education 
have higher postprogram test scores simply because they are more motivated. 

One way to gain perspective on the magnitude of these “effects” is to compare them with 
the association between formal education and achievement among these individuals before their 
participation in adult education. To accomplish this, we estimated the association between the 
highest grade completed by sample members and their reading and math scores at the time of 
random assignment. An additional year of formal education is associated with 8.3 points, or .17 
of a standard deviation, on the baseline TALS test. Assuming that sample members attended 
school for an average of 9 months per year, multiplying this by two years gives an amount of 
prior education comparable to 18 months in adult education. According to this comparison, 18 
months of formal education among this group of people yielded an average of 16.6 points, or .34 
of a standard deviation, on the TALS. Compared to the predicted .40 standard deviation differ- 
ence associated with 18 months in adult education, this suggests that, among this group of indi- 
viduals, the payoff to adult education is similar to the average payoff to the formal education they 
received before entering the program. Therefore, for the sample we studied, the literacy payoffs 



21 Gujarati, 1995. 
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to formal education and adult education appear to be quite similar, or slightly larger than the pay- 
offs associated with the years they spent in formal education. 

With respect to math skills, the regression estimates suggest that a year of formal educa- 
tion was associated with 1.46 points, or .10 of a standard deviation, on baseline math scores. 
Again assuming again that each year of formal education was associated with about 9 months in 
school, this suggests that 18 months of formal education would be associated with a difference of 
about .21 of a standard deviation in math skills. The predicted difference in math skills between 1 
month and 18 months of adult education is 8.94 points, or .64 of a standard deviation 22 — more 
than three times the payoff to respondents 'prior formal education. 

The relationship between GED receipt and participation in adult education follows a more 
bell-shaped pattern during the first year and a half. There appears to be a positive association be- 
tween months of adult education and the percentage of individuals receiving their GED during 
the first six months. Then, for the next six months, the relationship appears to flatten out and be- 
come negative. The estimates in Table 3.5 suggest that 8 percent of the individuals with one 
month of adult education obtain their GED within the two-year follow-up period. On the other 
hand, 29 percent of the welfare recipients with six months of adult education obtained their de- 
gree during the follow-up period — a difference of 21 percentage points. In proportional terms, 
however, we can say that individuals with six months of adult education are 263 percent more 
likely to receive their GED than individuals with only a minimal amount of participation. 

The results presented in this chapter also show conclusively that, in terms of test scores, 
the payoff to adult education was unrelated to whether or not it occurred under a mandate. On the 
other hand, the relationship between GED receipt and months of participation in adult education 
was positively related to whether or not that participation occurred as part of a mandatory wel- 
fare-to-work program. Thus, this evidence does not support concerns that mandated participation 
in adult education will not yield improvements in educational attainment and achievement. 

The evidence here suggests that, whereas all subgroups of welfare recipients in our sam- 
ple improved their measured skills after participating in adult education, for certain subgroups 
there was no relationship between adult education and GED receipt. In particular, those who en- 
tered the program with less than an eighth grade education experienced no relationship between 
time spent in adult education and GED receipt during our two-year follow-up period. This is con- 
sistent with program effects presented in Chapter 2 and with the idea that these individuals need 
so much remediation that it is unlikely that short-term participation in adult education will help 
them obtain a GED. On the other hand, although the differences are not statistically significant, 
the estimates in this chapter suggest that, if anything, these individuals experienced a stronger 
relationship between time spent in adult education and postprogram academic achievement. 

The evidence suggests that, to the extent that welfare-to-work programs are interested in 
achieving improvements in literacy and math skills among individuals who do not possess a high 
school diploma or GED, they may want to target referrals toward students with the least formal 
education. On the other hand, to the extent that they are interested in producing differences in 

22 This difference was generated using the coefficients in Table 3.3, column 2. The predicted difference between 
1 and 18 months = 5*.78 + 6*.35 + 6*.49 = 8.94. 
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educational attainment among this group, they might consider targeting instruction toward stu- 
dents with higher levels of formal education. Moreover, this pattern suggests that the approach 
taken, the expectations held, and the time allotted for completion of a program of adult education 
ought to vary according to an individual’s initial preparation. The evidence also suggests that re- 
ported barriers to participation, such as family and personal problems or the presence of young 
children — although potential impediments to participation — are not significant impediments to 
an individual’s ability to benefit from participation. 

At the level of the agencies that provide training to adult education clients, only average 
teacher experience and the proportion of teachers with advanced degrees at each provider ap- 
peared to be related to the payoff to additional time spent in adult education. In particular, stu- 
dents appeared to experience larger improvements in literacy scores when they attended classes 
at providers with more experienced and better educated teachers. This suggests that to help cli- 
ents improve these skills, welfare programs should consider referring students to providers with 
more qualified and more experienced teachers. On the other hand, to the extent that they are in- 
terested primarily in helping clients to obtain educational credentials, a focus on teachers’ educa- 
tional qualifications might not make a great deal of difference. 

However, these findings should be interpreted with some caution. The provider-level 
characteristics in this study were measured in a way that failed to capture the specific experiences 
of individual students. As a result, they may underestimate the effects of these factors on the 
payoff to time spent in adult education. To the extent that this is a concern, future research should 
attempt to gather data that more specifically capture the circumstances, experiences, and educa- 
tional resources available to each student. 



Chapter 4 



Does the Low-Wage Labor Market Value Basic Education? 
Effects of GEB Receipt and Literacy Gains on the Self-Sufficiency 

of Welfare Recipients 



I. Imtrodictkm 



The purpose of this chapter is to analyze how milestones and benchmarks used in adult 
education programs relate to subsequent employment and self-sufficiency outcomes. The previ- 
ous chapters discussed how adult education programs being operated in the context of state wel- 
fare-to-work initiatives affect traditional education outcomes, such as the receipt of high school 
credentials and the improvement of basic reading and math skills. An important assumption un- 
derlying the use of these outcomes is that they are related to subsequent success in the labor mar- 
ket. It is assumed that welfare recipients who increase their educational attainment and improve 
their basic skills will be better able to secure and retain employment and will be offered higher 
wages, enabling them to leave public assistance and become self-sufficient. 

There is an extensive literature to support these basic assumptions. For decades, labor 
economists and education researchers have documented how accumulated education, basic skills, 
and education credentials affect earnings and other employment outcomes. 1 Since the early 
1990s, researchers have found that the importance of education and skills in the labor market has 
increased, which partly explains the growing earnings inequality in the U.S. labor market during 
the last two decades. 2 All this is of particular concern to welfare recipients, whose basic skills are 
known to be lower than those of workers in the general population, 3 and whose prospects in the 
labor market are limited by this lack of skills and the related lack of educational credentials and 
work experience. 4 

Within the welfare population, basic skills and education credentials are important predic- 
tors of the extent of recipients’ long-term welfare dependency. Pavetti 5 and Bane and Ellwood 6 
performed analyses of welfare dynamics over time and found that recipients of AFDC who had 
lower levels of education were predicted to remain on welfare for longer periods. 

Given all this, welfare-to-work programs have often included education and training 
components to increase the basic skills of welfare recipients. Also, education was an important 
component of the federal Job Opportunities and Basic Skills Training (JOBS) program, the cen- 
terpiece of the 1988 Family Support Act. However, many welfare-to-work programs use other 
approaches, focusing not on the classroom but on quick job entry and developing of skills in the 



'Mincer, 1974; Polachek and Siebert, 1993; Sumet al., 1995. 

2 Levy and Mumane, 1992; Burtless, 1994. 

3 Kirsch et al., 1993; Zill et al., 1991. 

4 Burtless, 1997; Olson and Pavetti, 1996. 

5 Pavetti, 1993. 

6 Bane and Ellwood, 1994. 



workplace. Gueron and Pauly 7 summarized evaluations of early state welfare programs using 
these different approaches. They compared “high-cost” and “low-cost” programs, with the former 
more likely to include education services. Their review did not find that high-cost programs were 
more effective than low-cost programs in terms of the employment outcomes of welfare recipi- 
ents. Some high-cost, education-focused programs were quite successful, but so were some pro- 
grams that were focused on quick job entry. 

Another important study in this regard, the evaluation of California’s Greater Avenues for 
Independence (GAIN) program, also failed to bolster Human Capital Development (HCD) pro- 
grams as a promising way to help welfare recipients become self-sufficient . 8 Of the six California 
counties included in the study, one was far more successful in operating its GAIN program than 
the others. Compared with the other five counties in the study, Riverside County ran a program 
that was focused more on Labor Force Attachment and that placed less value on adult education 
and attainment of an education credential. 

Within the GAIN evaluation, a special study was conducted of the effects of GAIN on 
those without high school credentials or otherwise deemed “in need of basic education.” This 
study found that GAIN increased the basic skills of these welfare recipients in San Diego but not 
in the other counties . 9 It found significant impacts on GED receipt in four out of five of the coun- 
ties. However, within the two-year follow-up period of this study, earnings and welfare impacts 
did not parallel the impacts on educational attainment and basic skills. Later nonexperimental 
work by Friedlander et al. failed to find such a relationship even in the third year of follow-up . 10 

The latest contribution to our understanding of HCD programs for welfare recipients is the 
National Evaluation of Wei fare-to- Work Strategies (NEWWS) — the study of which this report is a 
part. As will be discussed in more detail in a later chapter, the NEWWS research design used ran- 
dom assignment to create experimental distinctions between two program approaches like those 
distinguished by Gueron and Pauly: Human Capital Development and Labor Force Attachment 
(LFA). Using random assignment both shields comparisons of these approaches from cross-site 
variation in program operation, welfare rules, and the social and economic environment and leaves 
the program approach as the only remaining factor that accounts for variation in program effects. 
Two-year findings from the NEWWS Evaluation were presented by Freedman et al., who enhanced 
the two-dimensional HCD-LFA picture by introducing other important dimensions, such as the 
programs’ levels of enforcement." Again, LFA programs appeared to be more successful than 
HCD programs, even in assisting welfare recipients who lacked a high school credential . 12 

Thus, the literature offers a somewhat confusing picture. On the one hand, researchers 
consistently find that basic skills and education credentials are important in the labor market, that 



7 Gueron and Pauly, 1991. 

8 Riccio et al., 1994. 

’Martinson and Friedlander, 1994. 

10 Friedlander et al., 1994. 

"Freedman etal., 1999. 

"Across the board, the most successful program studied was the JOBS program in Portland, Oregon, which of- 
fered a more varied approach, combining elements from traditional LFA and HCD programs. However, it is difficult 
to separate the benefits of this approach from other characteristics of the Portland program and its local environment 
(see Scrivener et al., 1998, pp. 13-43, for an extensive discussion of the features of this program). 
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persons who lack these assets are more likely to be poor and dependent on welfare, and that wel- 
fare recipients with lower basic skills remain on welfare longer. However, programs that directly 
address these shortcomings in clients’ human capital are no more successful than programs that 
do not and that focus on quick job entry instead. What might explain this discrepancy? 

Four important issues play a role in this discussion — all of which have been raised in ear- 
lier chapters in this report. First, the “human capital” literature focuses on skills and education 
accumulated during a lifetime, while the welfare program evaluation literature focuses on the ef- 
fects of relatively short-term and low-intensity programs that are remedial in nature. A credential 
obtained in one context may not be interchangeable with a credential obtained in the other. 

Second, the HCD programs studied were mandatory in nature, and the adult education 
services provided in them were usually not targeted specifically at welfare recipients. Chapter 2 
of this report showed that many welfare recipients in the NEWWS programs did not want to go 
to school to study basic reading or math despite their lacking a high school credential and their 
low basic skill levels. Therefore, the evaluation literature measures the effects of education and 
training programs that are mandatory, unpopular, often not designed for the needs of welfare re- 
cipients, and unable to retain most students long enough to effect real change in their outcomes. 

Third, all evaluations of education and training programs, whether welfare-related or not, 
have follow-up periods that are quite short. Combined with the low intensity and short duration 
of the programs themselves, this makes it difficult to identify program effects if they occur, espe- 
cially as they affect more long-term outcomes such as job retention, earnings growth, and sus- 
tained self-sufficiency. 

And fourth, the “human capital” literature may to some extent capture systematic underly- 
ing differences in the characteristics and backgrounds of different populations with different lev- 
els of educational attainment. 

II. Summary of the Findings 

® Our analyses suggest that receipt of a GED improves the earnings of 
those assigned to participate in welfare-to-work programs. The estimated 
effect of a GED on annual earnings three years after random assignment 
was approximately $771. GED recipients also were predicted to receive 
fewer welfare benefits. 

Simple regression analyses suggest that welfare recipients benefit from receiving a GED certifi- 
cate. This finding is fairly robust, holding up across different samples and different specifications 
of the analytical models. However, we were unable to confirm this finding using advanced statis- 
tical methods. Consequently, this finding remains tentative, and there is always a possibility that 
it is affected by model misspecification or selection bias. 

© Estimated effects of the GED were unaffected by the introduction of par- 
ticipation variables or reading test scores into the regression model. This 
suggests that our estimates capture the “credential effect” of the GED, 
rather than underlying skills or participation in adult education. 



- 113 - 



129 



Apparent effects of educational credentials on earnings and welfare receipt can represent differ- 
ent aspects of sample members’ human capital. On the one hand, it is possible that such creden- 
tials “certify” individuals’ participation in basic education or mastery of basic skills. On the other 
hand, the effect of the credential could be more superficial, enhancing the holders’ ability to sig- 
nal her persistence, seriousness, or employability. Our analyses suggest that the latter is the pri- 
mary function of the GED for welfare recipients. Controlling for participation in education or test 
scores does not reduce estimated effects of the credential, as might be expected if the credential 
were simply a proxy for skills or classes taken. 

° The benefits off having a GED were greater when the credential was earned 
as part off a larger welffare-to-work program. 

Program group members who received a GED appeared to benefit more from this credential than 
did control group members. This is probably because after receiving the credential program 
group members would be more likely to proceed with a structured job search or with additional 
education or training, as mandated (and supported) by the welfare-to-work program. 

o Greater reading skill levels were associated with higher earnings, regard- 
less off GED receipt. 

In analyses linking reading skills to earnings and welfare receipt, an increase of one standard de- 
viation in TALS scores (measured at a two-year interview) was associated with $354 in addi- 
tional earnings; however, there was no concomitant reduction in welfare receipt a year later. 

III. Tlhie Vaflue of Mediatmg Outcomes 

One way to assess the effectiveness of HCD programs in general, and of adult education 
programs in particular, is to focus on these programs’ effects on more immediate benchmarks 
and milestones. In this report, such analyses were done in Chapters 2 and 3, which showed im- 
pacts on reading and math scores and on GED receipt. In the relationship between adult educa- 
tion and improved employment outcomes these education outcomes operate as mediating vari- 
ables. That is, effects of adult education programs on earnings and self-sufficiency are supposed 
to occur in a linear process like the path shown in Figure 4.1. 

Figure 4.1 

Hypothesized Relationship Between Admit Edmcatiom and 
Labor Market Outcomes 




The assumptions underlying HCD programs do not identify other ways in which program 
participation on the left side of the figure affects the positive change in employment outcomes on 



the right side of the figure. (There is the possibility of a direct negative effect, inasmuch as some 
adult education participants may temporarily forgo employment opportunities.) Therefore, if Fig- 
ure 4.1 is a valid depiction of reality, it should be possible to evaluate the effectiveness of these 
programs by concentrating on the boxes in the middle of the figure. This is both more convenient 
and more appropriate, because these outcomes can easily be collected by program administrators 
and do not suffer from the kind of delayed response affecting the link between participation and 
employment outcomes. However, this course of action raises a significant problem: its appropri- 
ateness depends entirely on the extent to which the mediating outcomes in the center of our 
model are truly predictive of the outcomes shown on the right. 

Another way to think about this is to break down the effectiveness of adult education pro- 
grams for welfare recipients into two parts. The first step concerns the programs’ ability to im- 
prove the mediating outcomes. This was discussed in the two preceding chapters. The second 
step concerns the extent to which improvements in these mediators actually translate into better 
employment outcomes and increased self-sufficiency. In other words, do the outcomes targeted 
by adult education programs make a real difference in welfare recipients’ ability to work and 
leave welfare? 

The literature focusing on these relationships in the context of adult education programs 
for welfare recipients is quite limited. Probably the most studied outcome of these programs is 
receipt of the General Educational Development (GED) credential. However, most GED studies 
have examined the effects of this credential for the larger population of GED holders, not spe- 
cifically as an outcome of adult education programs. In probably the most widely cited study of 
the labor market value of the GED, Cameron and Heckman found no significant difference in 
wages between GED holders and high school dropouts who did not have GEDs. 13 (Their findings 
were based on data from the National Longitudinal Study of Youth, or NLSY.) The only positive 
effect of GED receipt that they found is an increase in college attendance, but they noted that the 
college dropout rate was much higher for GED holders than for college students with regular 
high school diplomas. 

Mumane et al. also used NLSY data to produce inferences about the labor market value 
of the GED and to describe how receipt of a GED affected subsequent participation in vocational 
training, college, and the military. 14 In their 1994 paper, Mumane et al. found evidence that 
among dropouts GED recipients were more likely to enter vocational training or college. They 
speculated that access to vocational training would provide a mechanism through which receipt 
of a GED could have a long-term and lasting effect on the labor market success of GED recipi- 
ents (the “gatekeeper” effect). Some support for this hypothesis was presented in a subsequent 
paper, 15 which found that the long-mn earnings trajectory of GED recipients was changed by 
their receipt of the credential, despite short-term reductions in their earnings and labor force par- 
ticipation as they pursued further education and training. 



,3 Cameron and Heckman, 1993. 
,4 Mumane et al., 1994, 1995. 
,5 Mumane et.al., 1995. 



A problem with these national studies based on NLSY data is that they use heterogeneous 
samples of high school dropouts to make comparisons between GED holders and those who do not 
have the credential. Many dropouts who do not pursue a GED may have decided against doing so 
because they are already employed or may think or know that they have good employment opportu- 
nities without a high school credential (because of a family business, for example). Therefore these 
findings may not be relevant for welfare recipients mandated to participate in adult education be- 
cause of their failure to complete high school. The relevant question for adult education programs 
serving these welfare recipients is whether the GED is a meaningful milestone for them, not 
whether it improves the earnings prospects of high school dropouts in general. 

An exception to these studies using national data was the analysis of the effects of GED 
receipt on earnings conducted by Quint et al. in their evaluation of the New Chance program. 16 
New Chance, a demonstration program developed by MDRC in the mid-1980s, was targeted ex- 
clusively at young teen mothers who were receiving welfare and had dropped out of high school. 
The program increased substantially the number of these young women who held a GED, but it 
failed to produce significant earnings gains or reductions in welfare receipt. The authors ap- 
proached this discrepancy by looking more closely at the earnings trajectories of sample mem- 
bers who received a GED. They found that GED receipt without subsequent training did not lead 
to better employment outcomes but that GED receipt followed by training did. They also found 
that program participants who attended basic education classes without completing them with a 
GED actually suffered earnings losses, which, in the average, offset the gains for those who did 
complete. However, it is unclear to what extent these findings from the New Chance program 
hold up in the larger welfare-to-work context. New Chance was limited to teen parents, was vol- 
untary, and included young women who were highly motivated but also experienced a multitude 
of barriers to program participation and subsequent employment. It is unclear to what extent find- 
ings from this small-scale demonstration project can be generalized to the larger world of manda- 
tory welfare-to-work programs. 

While many studies have been done of the effects of obtaining education credentials, 
many fewer researchers have looked at the effects of improvements in basic skills. Mostly this is 
because it is quite difficult (and expensive) to measure changes in basic skills, especially in stud- 
ies that involve a control group not subject to the program intervention. Among the exceptions 
are the New Chance demonstration and the evaluation of California’s GAIN program. The 
New Chance demonstration found no program effects on literacy as measured with the Test for 
Adult Basic Education (TABE). Quint et al. did not explore the relationship between TABE 
scores and employment outcomes. In the GAIN evaluation, Martinson and Friedlander examined 
the impacts of California’s GAIN program on TALS reading scores. 19 They found no impacts in 
most of the counties studied, but significant gains occurred in San Diego County. In their discus- 
sion of these findings they argued that the intensity of participation in basic education had proba- 
bly not been large enough to produce consistently significant findings. Martinson and Friedlander 



l6 Quint et al., 1997. 

17 Quint et al., 1997; Quint et al., 1994. 
l8 Martinson and Friedlander, 1994; c Riccio et al., 1994. 

l9 Martinson and Friedlander, 1994. For detailed descriptions of the TALS and CASAS tests, please refer to Ap- 
pendix D. 
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did not study the relationship between test score gains in San Diego and subsequent employment 
and welfare outcomes there. 

IV. About the Amalyses Fresemted Here 



The purpose of the analyses presented in this chapter is to expand what we know about 
the effects of GED receipt and basic education for welfare recipients. Unlike the larger programs 
assessed in the main NEWWS Evaluation, these relationships cannot be studied directly using 
random assignment. It is simply not possible to “assign” individuals a certain number of hours of 
participation, to “assign” them a GED, or to randomly give them certain basic skill levels. Con- 
sequently, it is necessary to employ nonexperimental research methods to identify these relation- 
ships. In this regard, this study of adult education programs for welfare recipients is no different 
from most of the nonexperimental research discussed above. However, the NEWWS data have 
several important advantages over other available data, mostly pertaining to the relevance of the 
findings in a welfare-to-work context and to the availability of appropriate control variables. 

The data collected for the NEWWS Evaluation offer an excellent opportunity to study the 
effects of adult education and GED receipt in a welfare-to-work context. The data used here 
cover a wide range of different welfare-to-work programs in different economic and program en- 
vironments — increasing the relevance and representativeness of conclusions based on these data. 
In its focus on welfare recipients in mandatory programs, this study answers fundamentally dif- 
ferent questions than much of the research cited above. Those studies described the effects of the 
GED for school dropouts in general, whereas the NEWWS data document those effects for drop- 
outs who are welfare recipients in particular. Although this sample may be more disadvantaged, 
it has access to a more comprehensive set of services, and it faces a mandate to participate. All 
this could affect how adult education and GED receipt impact employment outcomes among wel- 
fare recipients. 

A second advantage of the NEWWS data is the rich array of control variables available. 
Two sets of variables collected at program intake are rarely available in other data sources. The 
first is a number of variables obtained from the Private Opinion Survey (POS). This survey, ad- 
ministered before sample members knew whether or not they would be enrolled in the local 
JOBS program, captured sample members’ opinions, preferences, and perceived barriers to par- 
ticipation and employment. Having these variables allows us to hold constant sample members’ 
motivation and their readiness to participate in adult education, to pursue an education credential, 
to look for work, and to leave public assistance. The elusive concepts of motivation and work- 
readiness — captured by this baseline survey — play an important role in the process of self- 
selection that usually distinguishes GED recipients from nonrecipients, high school graduates 
from dropouts, and employed people from those who are not working. In most cross-sectional 
analyses (and, indeed, most time-series analyses as well) these concepts are difficult to control 
for. This means that estimates of effects of the GED, adult education, and other predictors of la- 
bor market success can be confounded with this uncontrolled self-selection process, exposing 
those estimates to what is commonly known as “selection bias.” 

Such bias can operate in various ways. Usually, it is assumed that if motivation and par- 
ticipation are confounded, estimates of program effects will be biased upward, that is, will seem 
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larger than they really are. However, as was suggested above, the selection process does not al- 
ways operate this way. Sometimes, motivation to participate in adult education or to pursue a 
GED may signal negative selection bias — for example, when a perceived lack of opportunity in 
the labor market prompts school dropouts to try to earn an alternative credential. Because this 
implies that GED recipients may sometimes have lower initial employment opportunities than 
their non-GED counterparts, estimates of the effect of the credential could be biased downward. 

Either way, inclusion of variables measuring motivation and work-readiness should sub- 
stantially minimize the extent of these selection biases. This will improve the quality of the esti- 
mates obtained, both in terms of their size and direction and in terms of their statistical precision. 

Another set of control variables available in the NEWWS data is a group of variables 
measuring sample members’ literacy and math skills before entry into a program. Many other data 
sources that are used to assess effects of adult education and GED receipt do not have such vari- 
ables, adding another potential source of selection bias to estimates based on these data. Again, the 
usual assumption is that such a bias would increase the apparent effects of adult education and GED 
receipt, because dropouts with higher initial achievement levels are more likely to pass a GED test. 
However, in this case, selection may work backwards as well. School dropouts with lower literacy 
levels may have more difficulty finding employment without remedial education or a GED certifi- 
cate, prompting them to enroll in adult education and pursue such a credential. Thus, not controlling 
for initial achievement levels also could produce either over- or underestimates of the effects of 
adult education and the GED. In seven of the eleven programs studied, the NEWWS data allow us 
to control for initial literacy and math scores, resolving this problem. 

A final advantage of the NEWWS data is that we may use unbiased experimental impacts 
on both employment outcomes and explanatory variables to check the consistency of our findings 
with the unbiased experimental impact story told elsewhere in this and other NEWWS reports. 
Impacts on GED receipt and test scores were presented in Chapter 2, and impacts on employ- 
ment, earnings, and welfare receipt were presented in previous NEWWS reports (for example, 
Freedman et ah, 1999). Consistent patterns among these experimental estimates would suggest 
that they might be related. This would help support findings of similar relationships produced 
with less reliable nonexperimental methods. 

In the remainder of this chapter, we will present effects of GED receipt and educational 
achievement gains on employment and welfare receipt of NEWWS sample members who did not 
have a high school diploma or GED at random assignment. 20 Details on the sample composition 
and sample sizes are provided in Appendix A. 

Table 4.1 describes the samples featured in this chapter in some detail, showing demo- 

2 1 

graphic characteristics for sample members in all 11 NEWWS programs. The table shows char- 



20 Note that some sample members in Riverside, California, did have a high school diploma or GED but were 
deemed “in need of basic education” because of their low initial achievement levels as measured with the CASAS 
test of basic skills. These sample members are excluded from analyses of the effects of GED receipt, but they are 
included in other analyses. 

21 These and other statistics presented in this chapter were weighted so that each of the seven NEWWS sites is 
given equal representation regardless of the actual size of the study sample in that site. 





National Evaluation of Welfare-to-Work Strategies 
Table 4.1 

For Sample Members Without a High School Diploma or GED at Random Assignment: 
Selected Characteristics off Sample Members, Pooled Across 11 Programs, 
by High School Diploma or GED Status at Follow-Up 





Received High School 


Did Not Receive High 






Diploma or GED at 


School Diploma or GED 




Characteristic 


Follow-Up 3 


by Follow-Up 3 


p-Value 



Demographic Characteristics 



Gender (%) 



Male 


1.3 


3 g #** 


0.001 


Female 


98.8 


2 #** 


0.001 


Age (%) 


Less than 19 


16.2 


4 Q *** 


0.001 


19-24 


31.5 


2\ 5 **# 


0.001 


25-34 


37.4 


46.2 *** 


0.001 


35-44 


14.5 


22 \ #** 


0.001 


45 and over 


0.4 


6 2 *** 


0.001 


Average age (years) 


26.2 


30.8 *** 


0.000 


Ethnicity (%) 


White 


51.0 


4 *** 


0.001 


Hispanic 


6.1 


g 7 **# 


0.001 


Black 


40.0 


47 2 


0.001 


Black Hispanic 


0.8 


0.1 ** 


0.001 


Native American/Alaskan Native 


1.3 


1.8 ** 


0.038 


Asian/Pacific Islander 


0.5 


0.7 


0.224 


Other 


0.4 


0.1 ** 


0.001 


Family Status 
Marital status (%) 


Never married 


64.1 


5 Q Q #** 


0.001 


Married, living with spouse 


1.5 


4 4 *** 


0.001 


Separated 


22.5 


25.0 *** 


0.003 


Divorced 


11.8 


lg 4 *** 


0.001 


Widowed 


0.1 


1 2 *** 


0.001 


Number of children (%) 


1 


44.7 


3 g ^ *** 


0.001 


2 


31.5 


30.4 


0.233 


3 or more 


23.9 


31 4 *** 


0.001 


Average number of children 


2.0 


2 \ *** 


0.000 


Has any child age 0-5 


80.7 


54 7 *** 


0.001 


Has any child age 6-1 1 


46.6 


51.6 *** 


0.001 


Has any child age 12-18 


19.7 


38.8 *** 


0.001 



(continued) 
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Table 4.1 (comtimued) 



Characteristic 


Received High School 
Diploma or GED at 
Follow-Up 3 


Did Not Receive High 
School Diploma or GED 
by Follow-Up 3 


p-Value 


Age of youngest child (%) 


2 or under 


45.0 


23.1 *** 


0.001 


3 to 5 


35.7 


31.5 *** 


0.001 


6 or over 


19.3 


45.3 *** 


0.001 


Had a child as a teenager 


61.4 


50.2 *** 


0.001 


Labor Force Status 

Worked full time for 6 months or more 


for one employer (%) 


48.4 


52.8 *** 


0.001 


Any earnings in past 1 2 months (%) 


32.1 


30.7 


0.113 


Currently employed (%) 


6.4 


7.2 


0.149 


Education Status 


Average highest grade completed 


10.2 


10.0 ** 


0.028 


Enrolled in education or training 


in past 12 months (%) 


34.6 


17.8 *** 


0.001 


Currently enrolled in education 


or training (%) 


37.1 


12 ^ 


0.001 


Public Assistance Status 
Total prior AFDC receipt 13 (%) 


None 


14.4 


9 q *** 


0.001 


Less than 1 year 


4.4 


2 q *** 


0.001 


1 year or more but less than 2 years 


12.5 


9 | *** 


0.001 


2 years or more but less than 5 years 


28.6 


26.7 ** 


0.030 


5 years or more but less than 1 0 years 


24.0 


20.7 *** 


0.001 


10 years or more 


7.2 


20.5 *** 


0.001 


Raised as a child in a household 


receiving AFDC (%) 


34.1 


33.5 


0.509 


First spell of AFDC receipt (%) 


20.3 


16.3 *** 


0.001 


Housing Status 
Current housing status (%) 


Public housing 


5.0 


12 2 *** 


0.001 


Subsidized housing 


14.1 


13.4 *** 


0.001 


Emergency or temporary housing 


5.2 


2 7 *** 


0.001 


None of the above 


76.6 


73.0 


0.268 



(continued) 
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Table 41 (continued) 



Characteristic 


Received High School 
Diploma or GED at 
Follow-Up 3 


Did Not Receive High 
School Diploma or GED 
by Follow-Up 3 


p-Value 


Site 








Atlanta 


5.0 


15.7 *** 


0.001 


Columbus 


13.0 


15.0 *** 


0.006 


Detroit 


23.0 


14 1 


0.001 


Grand Rapids 


15.8 


14.7 


0.145 


Oklahoma 


18.1 


14.5 *** 


0.001 


Portland 


18.2 


14.5 *** 


0.001 


Riverside 


4.4 


9 2 *** 


0.001 


Sample size 


280 


3,994 





SOURCE: MDRC calculations from information routinely collected by welfare staff. 

NOTES: To compensate for differences in sample members’ likelihood of being chosen to be surveyed, 
respondents were weighted by the inverse of the probability of their being chosen. In addition, each site was 
weighted equally in the pooled estimates. 

Distributions may not add to 1 00 percent because of rounding. 

a The GED credential is given to those who pass the GED test and is intended to signify knowledge of high 
school subjects. 

‘This refers to the total number of months accumulated from one spell or more on an individual's own or 
spouse's AFDC case. It does not include AFDC receipt under a parent's name. 

‘This does not mean that such individuals are new to the AFDC rolls, only that this is their first spell on 
AFDC. This spell, however, may have lasted several years. 
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acteristics for program group members and control group members combined. It breaks down 
these characteristics by educational attainment status at follow-up, showing how those who re- 
ceived a credential differed from those who did not. Asterisks and p-values in the table show 
whether differences across the two groups are statistically significant. In general, it appears that 
GED recipients and those who received a high school diploma were younger than those who did 
not receive an education credential. Those with a credential were significantly more likely to be 
younger than 25, were more likely to be never married, were much more likely to have young 
children (more than 80 percent had a child under 6), but were somewhat less likely to have been 
employed lull time for six months. All these differences suggest that GED recipients were less 
work-ready than their counterparts who did not get such a credential. On the other hand, GED 
recipients had less long-term welfare dependency, were more likely to have been enrolled in edu- 
cation or training programs in the past year, and were less likely to be living in public housing. 

V. Data Sources amd Outcome Measures 

In our analyses, we combined a number of different data sources. Our primary explana- 
tory variables (GED receipt, literacy, and math skills) were measured using a two-year follow-up 
survey conducted in each of the seven NEWWS sites. 22 This survey was conducted long enough 
after random assignment that most sample members would have completed their participation in 
adult education programs and most GED recipients would have obtained that credential by then. 
However, two years would have been too short to measure the effects of participation in educa- 
tion and GED receipt on sample members’ employment outcomes and self-sufficiency. There- 
fore, we used three-year data from unemployment insurance earnings records and state or county 
welfare records to measure those effects. To keep them simple, our analyses are usually limited 
to five distinct outcomes: 

® total eamings in year 3 

® increase in eamings from year 2 to year 3 

© average eamings per quarter employed in year 3 

® number of months receiving AFDC in year 3 

® change from year 2 to year 3 in the number of months receiving AFDC 

In addition to the outcome variables and the explanatory variables from the survey, the 
analyses use control variables from baseline enrollment forms, the Private Opinion Survey con- 
ducted at baseline in four of the sites, baseline achievement tests, and administrative data cover- 
ing the year before random assignment. 



22 As indicated earlier, reading and math tests were conducted in only three of the seven sites. 
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VI, The Vatae of the GED: Evidence from Experimental Comparisons 

As mentioned above, a rudimentary way to examine the relationship between GED re- 
ceipt and employment and welfare outcomes is to compare experimental impacts on these out- 
comes for various sites and programs. The question is whether patterns of these impacts would 
support the hypothesis that GED receipt improves employment and welfare outcomes. (That is, 
were strong impacts on GED receipt observed for the same programs for which we found strong 
impacts on earnings or welfare receipt?) 

Figure 4.2 shows fully experimental impact estimates for each of the 1 1 programs studied in 
the NEWWS Evaluation. 23 Impacts on GED receipt are shown on the horizontal axis, and impacts 
on year 3 earnings and year 3 AFDC receipt (in dollars) are shown on the vertical axis. (The earn- 
ings impacts are marked with diamonds; the AFDC impacts, with circles.) Looking at the earnings 
impacts, it seems that, with the exception of the Grand Rapids LFA program (the outlier on the left) 
and the Traditional program in Columbus (the only negative earnings impact), there is a modest 
positive relationship between impacts on GED receipt and impacts on earnings in the 1 1 programs 
studied. That is, the pattern of earnings impacts slopes up to the right of the graph. 

However, such a relationship does not seem to exist in the AFDC impacts (all of which 
were negative, as shown in the graph). The reason for this may be that two opposing processes 
account for the impacts on AFDC receipt. On the one hand, AFDC payments could go down be- 
cause of sanctions for nonparticipation — an effect that would probably be larger in programs 
with smaller impacts on GED receipt. On the other hand, however, one would expect AFDC im- 
pacts to follow earnings impacts, showing lower payments for sample members with higher earn- 
ings. This would result in larger AFDC reductions for programs with larger impacts on GED re- 
ceipt. Clearly, Figure 4.2 does not allow us to distinguish between these two explanations. 

VII. The Value of the GED: Ordinary Least Squares 
Regression Analysis 

The next step in our analysis was to compare the earnings, earnings growth, and welfare 
receipt of GED recipients directly with the outcomes for sample members who did not receive a 
GED. To do this, we use a straightforward statistical procedure called ordinary least squares 
(OLS) regression analysis. In such a procedure, the effects of explanatory variables on outcome 
variables are measured by examining their relationship while simultaneously holding constant 
many other determinants of the outcome variables. This is done by introducing measures of those 
other determinants as “control variables” in the analysis. Common examples of such control 
variables are age, household composition, and prior work experience (see box). 

As indicated above, however, the NEWWS data cover much more than the usual set of 
control variables, extending to important concepts such as motivation, readiness for school or 
work, literacy, and math skills. The analysis also includes a regular JOBS variable, which cap- 
tures other aspects of the JOBS program besides the explanatory variables of interest. In the ab- 
sence of such a variable, effects of such program components as case management, sanctions. 



23 Remember, these are Atlanta HCD, Atlanta LFA, Columbus Integrated Case Management, Columbus Tradi- 
tional Case Management, Detroit, Grand Rapids HCD, Grand Rapids LFA, Oklahoma City, Portland Oregon, River- 
side HCD, and Riverside LFA. 
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Figure 4.2 

For Sample Members Without a High School Diploma or GED at Random Assignment: 
Program Impacts on Earnings and AFDC Receipt in Year 3, 
by Impact on High School Diploma or GED Status After Two Years 
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SOURCES: MDRC calculations from unemployment insurance (UI) earnings records, AFDC records, and the JOBS 
Two-Year Client Survey. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment 
characteristics of sample members. 

To compensate for differences in sample members’ likelihood of being chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally in the 
pooled estimates. 
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and job search assistance could be attributed inadvertently to the education variables that are of 
interest to us. 

A. The Basic Model 

Table 4.2 presents ordinary least squares (OLS) estimates of the effects of getting a 
GED (during the two-year follow-up period) on selected employment, earnings, and welfare 
outcomes in year 3. These are “average” effects for all sample members in the study, regard- 
less of their research group status. The table shows that receiving a GED had substantial posi- 
tive effects on welfare recipients’ earnings, increasing those earnings by $836 on average in 
the third year of follow-up. This represents an increase of 30.5 percent of the mean for this 
outcome for the full sample. 



Standard Coratrol Variables 

Analyses featured in this chapter always use the same set of basic control variables. 
The purpose of these variables is to isolate the effects of GED receipt, participation in 
adult education, and variation in basic skill levels even if those are correlated with 
other variables, such as demographics, motivation, or welfare history. The list of 
standard control variables follows: 



Site (city, state) 

Assignment to Job Opportunities and Basic Skills Training (JOBS) program 

Marital status 

Number of children 

Any child older than 5 

Ethnic group 

Age 

Gender 

Employed in prior year/prior quarter 

Earnings in prior year (also squared)/prior quarter 

Received AFDC/Food Stamps in prior year/prior quarter 

Number of months received AFDC/Food Stamps in prior year 

Average amount of AFDC/Food Stamps per month in prior year 

Parental concerns about going to work or school 

Preference for school 

Extent of family/personal problems 

Feelings of depression 

Feelings of mastery over life events 



The table also shows that this increase is accompanied by a substantial positive impact on 
sample members’ earnings growth since the second year of follow-up. The mean difference in 



National Evaluation of Welfare-to-Work Strategies 

Table 4.2 



For Sample Members Without a High School Diploma or GED at Random Assignment: 
Estimated Effects of Receiving a GED During Years 1-2 on Selected Year 3 Outcomes, 

Pooled Across 11 Programs 



Outcome 


Mean 

Outcome 


Estimated Effect 
of Receiving a GED a 


p- Value 


R 2 


Earnings in year 3 ($) 


2,741 


+836 *** 


0.001 


0.152 


Increase in earnings relative to 
year 2 ($) 


546 


+797 *** 


0.000 


0.036 


Earnings per quarter worked in 
year 3 b ($) 


1,615 


+128 


0.179 


0.169 


Total AFDC received in year 3 ($) 


2,620 


-237 * 


0.059 


0.298 


Number of months receiving 
AFDC 


6.50 


-0.13 


0.640 


0.174 


Change in number of months 
receiving AFDC, compared to 
year 2 


-1.36 


-0.73 *** 


0.001 


0.034 


Sample size 


4,274 









SOURCES: MDRC calculations from unemployment insurance (UI) earnings records, AFDC records, Private Opinion 
Survey data, and the JOBS Two-Year Client Survey. 



NOTES: To compensate for differences in sample members' likelihood of being chosen to be surveyed, respondents were 
weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally in the pooled 
estimates. 

Results presented in this table were obtained from ordinary least squares regression analyses, controlling for baseline 
characteristics, research group (program or control group), and barriers and attitudes measured with the Private Opinion 
Survey. 

A two-tailed t-test was applied to estimated regression coefficients. Statistical significance levels are indicated as: * = 
10 percent; ** = 5 percent; and *** = 1 percent. 

‘‘The GED credential is given to those who pass the GED test and is intended to signify knowledge of high school 
subjects. 

hThis outcome is available only for those with any measured employment during year 3. Consequently, the size of the 
sample (2,217) is less than shown at the bottom of the table, and estimates may not be as reliable as those shown in other 
rows of the table. 
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earnings between the second and third years of follow-up was $546, but for those who received a 
GED during the follow-up period, this earnings growth measure was $797 higher than it would 
have been had they not earned this credential. This suggests that most of the effect of GED re- 
ceipt on earnings took until the third year of follow-up to materialize, which is consistent with 
the theory underlying HCD programs. However, contrary to what one might have expected, GED 
receipt did not significantly increase average earnings per quarter worked. 

As Chapter 2 showed, welfare-to-work program group members were more likely than 
control group members to earn a GED during the two-year follow-up period. This means that 
GED receipt and program assignment are correlated, necessitating the inclusion of a separate 
variable in the analysis to account for other aspects of the welfare-to-work programs. Without 
such a variable, the effects of case management, sanctions, or other program features might have 
been attributed to the GED. The coefficients on this program variable (not shown in the table) 
were almost as large as the coefficients associated with GED receipt. Being in a welfare-to-work 
program increased year 3 earnings by $796, earnings growth by $447, and earnings per quarter by 
$221 — all statistically significant at the .01 level. This suggests that receipt of a GED certificate 
was by no means the only way by which program group members benefited from their participa- 
tion in the welfare-to-work programs. 

Having a GED also reduced AFDC benefits in year 3, by $237, or approximately 10 per- 
cent of the sample mean. Again, the strongest effect concerned the change in AFDC receipt since 
the second year of follow-up. The average sample member received AFDC in 1.36 fewer months 
in year 3 than they had in year 2. Receipt of a GED further reduced this by 0.73 months. 

B. Adding Participation Variables 

Analyzing the effects of GED receipt, one might ask whether these effects are driven 
purely by the newly obtained credential or whether they reflect educational gains from participa- 
tion in adult education programs. Analyses presented in Table 4.3 address this issue by introduc- 
ing two participation variables into the ordinary least squares regressions whose results were re- 
ported in Table 4.2. These variables, obtained from the two-year survey, measure (1) whether 
sample members ever participated in basic education and (2) how many months they participated. 
The effects associated with a unit change in each of these variables are presented in Table 4.3, 
together with the effects of GED receipt. 

It is noteworthy that the estimated effects of GED receipt did not change markedly with 
the addition of the participation variables in Table 4.3. This suggests that the benefits of receiv- 
ing the GED were mostly unrelated to the education provided in welfare-to-work programs. This 
does not mean that participation in adult education has no measurable benefits. Instead, it seems 
that the effects of GED receipt and the effects of participation in adult education were comple- 
mentary. For purposes of interpretation, this means that the GED effect appeared to represent a 
“credential” effect, more than a “certification” effect. In other words, rather than certifying suc- 
cessful completion of a course of study, the credential attested to the holder’s command of key 
skills and knowledge. If the reverse had been true, and the GED would primarily have signaled 
participation in a remedial education program, the estimated GED effect should have been re- 
duced upon inclusion of the participation variables in the regression. Note that this distinction 
does not address the question of whether the skills measured by the GED are valuable in the la- 
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bor market or whether something else about the credential (for example, its “sheepskin” effect) 
underlies its apparent payoff. A later section will address this issue. 

The participation effects shown in Table 4.3 followed an interesting pattern. Comparing 
the effects on year 3 earnings and year 3 earnings growth, it appeared that the effects of participa- 
tion in adult education were even more delayed than the earlier effects on GED receipt were. This 
makes sense, because participation in adult education programs takes time — more so than pursuit 
of a GED. As a result, participants may have been diverted temporarily from seeking work, 
which would have had the effect of diluting early improvements in employment outcomes. The 
estimated effects of additional months of participation on earnings was generally negative, albeit 
never statistically significant. It is unclear exactly how these two effects played out, given the 
participation patterns found in this sample, but the most beneficial course of action seems to have 
been brief enrollment in an adult education (or GED preparation) program, followed by attain- 
ment of a GED. Such a scenario would have been more positive than a much more lengthy spell 
of adult education without a credential. 

C. Separate Effects for Program and Control Group Members 

So far, the analyses presented have shown average effects for all sample members, 
regardless of their research group status. However, it is by no means certain that those assigned 
to one of the welfare-to-work programs would have experienced the same effects from getting a 
GED or participating in adult education than those assigned to a control group. In fact, there are 
two distinct processes that could produce different effects for those in welfare-to-work programs 
and those excluded from them. 

First, we know from Chapter 2 that program group members were more likely than con- 
trol group members to have earned a GED. In theory, one would expect this to be reflected in the 
characteristics of those who earned this credential. Presumably, control group members who de- 
cided to try to get a GED would have had to be more motivated to make this decision and to fol- 
low through on it than program group members who were offered assistance and incentives to do 
the same. In turn, this difference in underlying motivation should have manifested itself in the 
employment and welfare outcomes of the two groups of GED recipients. If bias from self- 
selection accounts for part of the observed GED effect, such bias should be greater among the 
control group, inasmuch as those GED recipients formed a more “selective” subsample than 
GED recipients in the program group. Such an effect should be especially strong in a mandatory 
welfare-to-work environment like the one studied here, in which GED recipients in the control 
group would have pursued the credential voluntarily, while GED recipients in the program group 
would have done so under pressure from program rules. Thus, even if selection bias were a rela- 
tively minor problem, one should see substantial differences in effects across the research groups. 

However, all this assumes that the underlying “true” effects of GED receipt and participa- 
tion in adult education were the same for both program and control group members, which does 
not have to be the case. It is possible that program group members benefited more from these 
services and credentials because they experienced them in the context of a larger program that 
could help them maximize the benefits of adult education and GED receipt. Program group 
members had access to postsecondary vocational training, job placement services, case manage- 



ment, and other supports — all of which could enhance the practical effect of a credential on 
someone’s employment outcomes and self-sufficiency. 

Table 4.4 supports the second of these two hypotheses. The estimated effects of GED re- 
ceipt were significantly larger for program group members than they were for controls, 24 suggest- 
ing that selection bias was not a serious problem and that a GED credential has stronger effects 
on employment outcomes when it is attained as part of a larger welfare-to-work intervention of- 
fering other supports in addition to GED preparation. 

B. Another Way to Capture Possible Selection Bias: Looking for Premature Effects 

Although the distribution of GED effects across program and control group members was 
reassuring, we devised another way to identify the potential effects of uncontrolled selection bias. 
We did so by estimating effects of the GED on earnings in the first year of follow-up, when most 
GED recipients did not have the credential for long (if they had earned it at all by then). A posi- 
tive effect of GED receipt on earnings during that year would have been suspicious, possibly 
identifying greater motivation, higher inherent skill levels, or other uncontrolled differences 
benefiting future GED holders. 

Carrying out this analysis (not shown in any tables), it appeared that our estimates more 
than met this test. The estimated effect of GED receipt on total earnings in the first year of follow- 
up was -$452, which is statistically significant at the .01 level. This suggests that future GED hold- 
ers were actually working less during the first year of follow-up, possibly because they were en- 
gaged in adult education programs instead of working in low-wage jobs. Controlling for participa- 
tion in adult education (as was done in the analyses presented in Table 4.3) reduced this estimate to 
-$289, which is still negative and statistically significant at the .10 level. In either case, the estimate 
was far from being significantly positive — another reassuring pattern of effects that supports our 
confidence in the longer-term benefits of GED receipt that were presented earlier. 

E. Instrumental Variables Estimates 

Another popular way to address the issue of selection bias in estimates like the ones pre- 
sented in this chapter is to use a statistical technique known as instrumental variables analysis. 



24 An F-test was conducted to determine whether the effects for program group members were statistically signifi- 
cantly larger than those found for controls. The p-value for this test (not shown in the table) was 0.0740, indicating a 
statistically significant difference. 

25 Specifically, instrumental variables estimation addresses bias that occurs when the error term in a regression 
model is correlated with one or more independent variables in the model. In the case of selection bias, such correlation 
occurs because an explanatory variable (such as GED receipt) represents an underlying selection process, which is not 
properly modeled in the regression equation. In the instrumental variables framework, a third variable, the “instrument,” 
is used to predict the explanatory variable and the resulting predicted value of that variable is used in the regression in- 
stead. For this procedure to be successful, the instrument must be highly correlated with the explanatory variable but 
uncorrelated with the outcome variable (except as mediated by the explanatory variable). In our case, we use the ran- 
domly created experimental treatment contrast, interacted with program site, as a source of such instruments. To ensure 
that the analysis does not misattribute program effects to the GED, we included other mediators — such as receipt of job 
search assistance, participation in basic education, and a proxy variable— to account for the immediate employment ef- 
fects of the mandate to participate in welfare-to-work programs. For more details and examples of this procedure, see, 
for example, Angrist, 1990; Angrist and Krueger, 1991; and Heckman and Vytlacil, 1998. 
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For the purpose of this analysis, we did not weight the sample as we did in all previous 
analyses, because instrumental variables estimates would have become sensitive to the experi- 
ences of small groups of sample members in individual sites and subgroups. (Also, we did not 
know how to judge the statistical properties of weighted instrumental variables estimates.) Table 
4.5 summarizes the results. 

Looking at the table, the estimates appear to be fairly consistent with those presented in 
earlier tables. However, Table 4.5 reveals how imprecise the instrumental variables method is, 
using these data. Estimates that were highly statistically significant in earlier tables now have 
large p-values, reflecting the large standard errors associated with them. These estimates are so 
imprecise because the relationship between the instruments used and the explanatory variables of 
interest is weak. Even though, as Chapter 2 showed, the NEWWS welfare-to-work programs 
caused significant increases in GED receipt, these effects (and their cross-site variation) were not 
large enough to make the experimental treatment variables adequate instruments for GED re- 
ceipt. On the other hand, these estimates did not contradict those presented earlier. 26 



VIII. The Importance off Basic Skills and Their Effects on Employment 

In addition to measuring effects of GED attainment, the NEWWS basic education data 
allow us to measure the effects of changes in basic skill levels, the second set of mediators pre- 
sented in the introduction to this chapter. As discussed earlier, data on follow-up reading skills 
were collected at the time of the two-year follow-up interview, using the TALS test to measure 
literacy. Not all survey respondents in all sites were administered these tests. The TALS data 
were available only for those assigned to HCD programs in Atlanta, Grand Rapids, and Riverside 
(and not in possession of a high school diploma or GED at baseline). The CASAS math scores 
were collected only for a subsample of this “TALS sample,” namely, those whose youngest child 
was older than 6 at baseline. 27 



A. Comparing GED Effects Across the Samples 

Because it is possible that average effects of GED receipt would be different in these two 
subsamples, we repeated the basic analyses presented in earlier tables for these samples. Table 
4.6 shows effects of GED receipt on the basic year 3 earnings and AFDC receipt measures for the 
full sample (the same as in Table 4.2), the TALS sample, and the CASAS sample. At $1,238 and 
$1,508, respectively, the earnings effects associated with GED receipt were larger in the TALS 
and CASAS samples than in the full sample (where we found an effect of $836). This may reflect 
the different composition of the TALS and CASAS samples (most notably, the fact that only 
three of the eleven NEWWS programs were represented), but it also underscores the sensitivity 
of the GED effects to sample selection and model specification. 



An interesting aspect of these analyses, not presented in Table 4.5, is the fact that these instrumental variables 
analyses uncovered effects of other mediating variables in the relationship between welfare-to-work programs and 
employment outcomes. The only mediator that was consistently statistically significant was participation in job 
search, which, for example, was associated with an increase of $1,834 in year 3 earnings — statistically significant at 
the 5 percent level. 

27 This distinction made it easier to administer a special survey module for parents of younger children. There is 
no substantive reason to exclude parents of younger children from the math test. 
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SOURCES: MDRC calculations from unemployment insurance (UI) earnings records, AFDC records, Private Opinion Survey data, and the JOBS 
Two-Year Client Survey. 

NOTES: Results presented in this table were obtained using instrumental variables analyses. Such analyses attempt to control for unmeasured 
differences between those who received a GED and those who did not (which might bias estimates from ordinary least squares analyses). 
Instruments were derived from the experimental (program-control group) contrast, created as part of the overall NEWWS Evaluation. Instrumental 
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Table 4.6 



For Sample Members Without a High School Diploma or GED at Random Assignment: 
Estimated Effects of Receiving a GED During Years 1-2 on Selected Year 3 Outcomes, 
by Availability of TAILS Document Literacy and CASAS Math Test Scores 



Outcome 


Mean 

Outcome 


Estimated Effect 
of Receiving a GED a 


p- Value 


R 2 


Full Sample 6 










Earnings in year 3 ($) 


2,741 


+836 *** 


0.001 


0.152 


Total AFDC received in year 3 ($) 


2,620 


-237 * 


0.059 


0.298 


Sample size 


4,274 








TALS Sample" 










Earnings in year 3 ($) 


2,652 


+ 1,238 *** 


0.002 


0.206 


Total AFDC received in year 3 ($) 


3,018 


-295 


0.157 


0.299 


Sample size 


2,296 








CASAS Sample* 1 










Earnings in year 3 ($) 


2,894 


+1,508 *** 


0.009 


0.179 


Total AFDC received in year 3 ($) 


2,501 


-161 


0.549 


0.279 


Sample size 


1,323 









SOURCES: MDRC calculations from unemployment insurance (UI) earnings records, AFDC records, Private 
Opinion Survey data, and the JOBS Two-Year Client Survey. 



NOTES: To compensate for differences in sample members’ likelihood of being chosen to be surveyed, respondents 
were weighted by the inverse of the probability of their being chosen. In addition, each site was weighted equally in 
the pooled estimates. 

Results presented in this table were obtained from ordinary least squares regression analyses, controlling for 
baseline characteristics, research group (program or control group), and barriers and attitudes measured with the 
Private Opinion Survey. 

A two-tailed t-test was applied to estimated regression coefficients. Statistical significance levels are indicated 
as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

TALS = Test of Applied Literacy Skills. 

CASAS = Comprehensive Adult Student Assessment System. 

“The GED credential is given to those who pass the GED test and is intended to signify knowledge of high 
school subjects. 

^he full sample includes all sample members in 1 1 NEWWS programs who did not have a high school diploma 
or GED certificate at random assignment and who responded to the Two-Year Client Survey. This is the same sample 
used in Tables 4.1 through 4.5. 

'The TALS sample includes all sample members in Atlanta, Grand Rapids, and Riverside who (1) were assigned 
to an education-focused program or to the control group, (2) did not have a high school diploma or GED certificate at 
random assignment, and (3) responded to the Two-Year Client Survey and completed the TALS test. 

d The CASAS sample includes all sample members in the TALS sample who also completed a CASAS math test. 
For logistic reasons, this test was not administered to sample members who had children between 3 and 6 years old. 
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B. Effects of VariatnoH in Literacy Skills 



In this section we present effects of basic literacy on employment, earnings, and welfare 
outcomes. There are three ways to look at these effects. First, we present these effects without 
controlling for educational attainment (GED status), which incorporates into the analysis the dif- 
ferences in skill levels between GED holders and non-GED holders. Then we introduce the GED 
into the analysis, which means that the estimates now capture the importance of basic literacy 
over and above the effect of having a GED. In a third version of the analysis, we add baseline test 
scores to the equation. This effectively limits the variation in literacy to measured literacy gains 
between baseline and follow-up. In theory this should not change the effects by much, but in 
practice the value of “preexisting” literacy skills may differ from that of newly acquired skills. 

Table 4.7 presents these three perspectives on the effects of literacy skills. 28 The top panel 
shows the effects of changes in TALS scores, without controlling for GED receipt or baseline test 
scores. The analyses found that if TALS scores were higher by a standard deviation, year 3 earn- 
ings would increase by $355 (or 13.4 percent of mean earnings), and earnings per quarter worked 
would be higher by $127 (8.0 percent of mean earnings per quarter). There were no comparable 
effects on earnings growth (since year 2) or AFDC receipt. The lack of a statistically significant 
effect on earnings growth may be explained by the fact that changes in basic skills are not imme- 
diate, as GED receipt is, which makes it unlikely that one would see a sudden change in the earn- 
ings or earnings trajectory of those who increase their skills through adult education. 

The second panel shows that controlling for GED status makes no significant difference 
in the size of most coefficients associated with literacy and math skills. All this reaffirms that the 
GED, as measured in this study, has primarily a “credential” effect, not representing fundamen- 
tally different skill levels but having an independent effect regardless of those underlying skills. 
This is consistent with other studies 29 that found that GED receipt was not accompanied by in- 
creased reading and math skills. All these findings, including those presented in Chapter 3 of this 
report, suggest that welfare recipients who receive a GED generally are close to being able to 
pass the GED test before they enter the programs preparing them. Additional skills taught in 
GED preparation classes apparently are not identified by basic skills tests such as TALS and 
CASAS. 

The third panel controls for baseline test scores, in addition to controlling for GED re- 
ceipt. In theory, this should not change the effects associated with variation in follow-up test 
scores, provided that skills acquired between baseline and follow-up are equivalent to skills ac- 
quired earlier. The only difference is that the variation in skills used in the analysis is more lim- 
ited, making the analysis less precise. However, the table seems to show otherwise, showing gen- 
erally smaller (and nonsignificant) coefficients associated with basic skills acquired after random 
assignment. One difficulty in interpreting this finding is that two-year follow-up test scores often 
were lower than initial test scores, possibly reflecting differences in the administration of the 



28 Note that all three analyses include the participation variables introduced in Table 4.3. Thus, apparent effects 
from literacy gains and GED receipt are not confounded with the education that produced them. 

29 Similar patterns were found by Martinson and Friedlander, 1994; Friedlander et al., 1994; and Quint et al 
1994. 
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Table 4.7 
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(continued) 



Estimated Effect of a Standard 
Estimated Effect Deviation Change in the 

Mean of Receiving a GED a TALS Literacy Score 

Outcome Outcome Effect p-Value Effect p-Value 
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tests. 30 However, it is quite possible that recent test score gains were not immediately reflected in 
sample members’ earnings and welfare outcomes. 

Last, we repeated the analyses presented in Table 4.7, adding CASAS math scores to 
the equation (not shown in tables). Doing so did not substantially change the estimated effects 
shown in Table 4.7. The coefficient on the CASAS variable itself was almost never statisti- 
cally significant. 

IX. Connclmsnoiiis 



This chapter examined how key educational outcomes of welfare-to-work programs relate 
to subsequent employment and welfare outcomes. We found some evidence that GED receipt is 
associated with higher earnings and lower welfare receipt, although we were unable to confirm 
these findings using more advanced statistical methods — mostly because NEWWS program ef- 
fects on GED receipt were too small. On the other hand, none of our analyses suggested that the 
personal characteristics and motivation of GED recipients were responsible for the apparent ef- 
fects of the credential. 

Our findings do not suggest that the effects of the GED represent either underlying effects 
of participation in basic education or underlying differences in basic skills. Both participation in 
basic education and increased basic skills have independent positive effects on earnings and earn- 
ings growth, although participation in basic education is also associated with longer welfare 
spells. 

All this suggests that receipt of education credentials and increases in basic skills are 
good indicators with which to assess the success of basic education programs for welfare recipi- 
ents. A focus on improving these mediators should result in better long-term outcomes for wel- 
fare recipients. However, future research should examine the effects of GED receipt in the con- 
text of welfare-to-work programs that have greater effects on this mediator. 

Finally, this chapter contributes some new lessons to the ongoing debate about the value 
of basic education and GED receipt for welfare recipients and other low-income populations. Our 
analyses suggest that welfare recipients do benefit from receiving a GED, supporting findings by 
Quint et al. (1997) and others. However, we were unable to reproduce our findings using instru- 
mental variables estimation, mostly because our estimates lacked precision. 



30 Follow-up tests were administered in sample members’ homes as part of a larger interview, whereas baseline 
tests were administered in the welfare-to-work program office, where there were fewer distractions and perhaps 
greater pressure to do well on the test. 
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Chapter 5 

A Comparison off Program Options for 
Welfare Recipients in Need off Basic Education 

I. Introduction 



The NEWWS Evaluation features 1 1 welfare-to-work programs in seven sites across the 
United States. Each program is being evaluated using a random-assignment research design, al- 
lowing us to reliably estimate the effects of the various programs on the people assigned to 
participate in them. However, besides learning about the effectiveness of each program, the fact 
that there are 1 1 different programs makes it possible to learn more. By systematically comparing 
program effects across all programs and sites, we can learn which approach may be most effec- 
tive with certain groups of welfare recipients. 

The purpose of this chapter is to describe and analyze the variation in program effects 
found in the NEWWS Evaluation, focusing on a subsample of the welfare recipients served by 
the NEWWS programs, namely, those lacking a high school diploma or GED. 1 It is hoped that 
such analysis will lead to greater understanding of which approaches are most successful with 
this population. 

Specifically, the analyses presented here examine variation in program effects along a 
range of program dimensions first introduced by Freedman et al. 2 In their analysis of two-year 
program effects for the full NEWWS sample (including those who did have a high school di- 
ploma or GED), Freedman et al. developed a typology of NEWWS programs, dividing them first 
by their level of enforcement and then by their substantive focus, which was considered either 
“employment-focused” or “education-focused.” In addition, Freedman et al. paid attention to 
each site’s sequencing of program activities and case management structure. The 1 1 programs 
featured in the NEWWS Evaluation were characterized along the dimensions shown in Figure 
5.1 (adapted from Freedman et al., 1999). 

Inasmuch as Freedman et al. presented effects for all these programs and separately dis- 
cussed the effects for participants without a high school diploma or GED, why are we revisiting 
these comparisons in this chapter? We do so for several reasons. 

First, Freedman et al. had only two years of follow-up data when they prepared their re- 
port. Although this is a long enough follow-up period to present general program effects, it may 
be too short to find effects for those lacking a high school diploma or GED. Especially when 
these sample members participate in basic education programs, it may take longer for positive 
impacts to materialize for them. At present, we have three years of follow-up data, allowing us to 
tell a more complete story about this subgroup of welfare recipients. 



'About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or 
GED, but they scored low on the math and reading portion of the appraisal test or had limited English and, thus, were 
determined to be in need of basic education. 

2 Freedman et al., 1999. 
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Figure 5.1 



Typology of NEWWS Programs 



Employment-Focused Approach 


Education-Focused Approach 


Job Search First Varied First Activity 


Education or Training First 


High Enforcement High Enforcement 


High Enforcement Low Enforcement 


Atlanta LFA Portland 

Grand Rapids LFA 
Riverside LFA 


Atlanta HCD Detroit 

Grand Rapid HCD Oklahoma City 

Riverside HCD 
Columbus Integrated 
Columbus Traditional 



NOTES: LFA = Labor Force Attachment; HCD = Human Capital Development. Programs in italics had 
integrated case management. 



Second, this chapter expands on the work of Freedman et al. and others in that it uses 
more advanced statistical techniques to sort out different causes of variation in impacts across 
different welfare-to-work programs. 3 While the dimensions shown in Figure 5.1 are our primary 
focus, differences in impacts across sites and programs can be attributed to many other dimen- 
sions, including variation in local environments, variation in sample characteristics, and aspects 
of program operations other than the ones highlighted. In many cases, what accounts for cross- 
site variation in program effects is not obvious, but this chapter will make an effort to find out. 

In addition to covering 1 1 programs in seven sites, the NEWWS Evaluation has a feature 
that not many evaluations of social welfare programs have: random assignment to multiple pro- 
grams in a single site. In Atlanta, Grand Rapids, Riverside, and Columbus, welfare recipients 
were assigned not just to a single NEWWS program but to one of two different programs (or to a 
control group). In the first three sites, the assignment was either to a Human Capital Develop- 
ment (HCD) program or to a Labor Force Attachment (LFA) program. 4 In Columbus, assignment 
was to a Traditional program or to an Integrated program featuring integrated case management. 
Because of random assignment, we can estimate fully experimental effects of assignment to one 
of these programs versus another, holding constant all other aspects of the program environment 
and sample composition. For analyses that go beyond this fully experimental framework, we in- 
stead have to rely on nonexperimental methods to separate programmatic differences from other 
factors that might interact with the effects of NEWWS programs. The large samples available in 
the NEWWS Evaluation make this study a good research environment in which to undertake 
such analyses. 

The structure of this chapter is straightforward. Section II presents key findings, and section 
HI introduces the methods used in the analysis; section IV describes the sample and the program 



3 For a discussion of the importance of understanding variation in effects across different program approaches 
see, for example, Greenberg and Wiseman, 1992; and Greenberg et al., 1993. For an application of some of the ap- 
proaches used in this chapter, see Bloom et al., 1993, 1997. 

4 See Chapter 2 for an overview of the two types of programs. 



environment. Section V then presents impacts for all 1 1 programs, section VI discusses differences 
in impacts across the programs, and section VII presents effects for a number of subgroups defined 
across the sample. Section VIII then offers conclusions. The purpose of this chapter is not to tell a 
comprehensive story about all relevant program effects for sample members entering without a high 
school diploma or GED. Instead, we focus on a few outcomes, attempting to identify program char- 
acteristics that best explain variation in effects concerning these outcomes. 

IS. Summary of the Findings 

© Generally, the welfare-to-work programs studied were successful 5u in- 
creasing sample members’ earnings and reducing their reliance on wel- 
fare. 

On average, welfare recipients who entered the 1 1 programs in the NEWWS Evaluation 
without a high school diploma or GED increased their three-year earnings by $1,212, or 21.7 
percent, and received $1,056 fewer welfare benefits (a reduction of 10.6 percent). Such favorable 
impacts were found for most of the individual programs and subgroups we studied. 

® Some significant differences in program effects across the 11 programs 
are attributable to differences in characteristics of the welfare recipients 
served by these programs. 

Although patterns of impacts across the 1 1 welfare-to-work programs were largely unaf- 
fected by our efforts to control for differences in individual characteristics across the sites, some 
programs — most notably those in Atlanta and Detroit — had more modest effects than the other 
programs, possibly because they served welfare recipients for whom such effects were more dif- 
ficult to achieve. 

o Experimental analyses show that education-focused programs may have 
smaller impacts than employment-focused programs, at least during the 
first two years of follow-up. Also, programs featuring integrated case 
management may have stronger impacts than programs with a tradi- 
tional separation of income maintenance and welfare-to-work tasks. 

The strongest comparisons featured in this chapter involve sites in which dual random as- 
signment created two separate program groups, each with its own program approach. In three 
sites — Atlanta, Grand Rapids, and Riverside — we studied education-focused and employment- 
focused programs side by side. Although impact estimates were stronger for the employment fo- 
cused programs in all three sites, the differences were statistically significant only in Grand Rap- 
ids. In that site, three-year program effects on earnings were more than twice as large for the em- 
ployment-focused LFA program than for the education-focused HCD program, although this 
difference was no longer statistically significant in the third year. Statistically significant differ- 
ences between program options were also found in Columbus, where the Integrated program had 
significantly greater earnings and AFDC impacts than the Traditional program. 

© Other comparisons found that programs with strong enforcement had 
greater effects on welfare receipt, but not on earnings. Also, Portland’s 
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flexible approach regarding participants’ first activity was more success- 
ful than other employment-focused approaches. 

After controlling for variation in individual characteristics across the sites, high enforce- 
ment programs remained more successful in terms of reducing AFDC receipt. For example, high 
enforcement education- focused programs reduced three-year AFDC receipt by $1,170, compared 
to $160 for similar programs without strong enforcement. Also, the program in Portland was 
consistently stronger than the other employment-focused programs (in Atlanta, Grand Rapids, 
and Riverside), even after controlling for differences in the populations served by these pro- 
grams. For example, the Portland program increased three-year earnings by $2,433, compared to 
$1,698 for the other three programs combined. 

o Tentative evidence from cross-site comparisons suggests that grant levels 
and unemployment rates are important site-level predictors of program 
effectiveness. 

We attempted to use variation in economic circumstances and the larger program envi- 
ronments in our data to identify predictors of program success other than those captured by nar- 
row program characteristics. Doing so resulted in some evidence that higher grant levels were 
associated with greater program effects and that higher unemployment rates were associated with 
smaller program effects, especially on earnings. 

o Subgroup analyses identified initial reading levels and personal barriers 
as important predictors of program success. Lower reading levels led to 
smaller program effects on earnings. Also, personal problems and other 
barriers significantly reduced program effectiveness, except in cases 
where these barriers were directly related to participants’ concerns about 
leaving their children to go to work or school. It is possible that program 
services may have mitigated the negative effects of the latter concerns. 

In our subgroup analyses, we focused on a narrow set of subgroup dimensions that were 
introduced in Chapter 2. Using only those, we identified low reading scores and self-reported 
personal or family problems as significant hurdles to favorable program effects. Sample members 
who, on the baseline Private Opinion Survey, indicated that they had personal or family problems 
but who did not have many concerns about leaving their children to go to work or school were 
the only subgroup studied that did not experience positive program effects on earnings. 

ME. Methods 



Figure 5.2 describes the relationship between assignment to a specific NEWWS program 
and subsequent program impacts. The NEWWS Evaluation is principally designed to answer 
questions about the thick horizontal arrow shown in the figure — that is, about the relationship 
between program assignment and program impacts. Understanding this relationship means that 
we know the effect of a specific program. 

However, researchers and policymakers often are interested in the other concepts illus- 
trated in Figure 5.2. For example, program developers may want to know how different pro- 
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grammatic factors contributed to the overall relationship between program assignment and pro- 
gram effects. In other words, they want to answer questions about the specific contributions of 
such program dimensions as the focus (education or employment), the level of enforcement (high 
or low) or the structure of case management (traditional or integrated). Others may want to learn 
more about how the program effects vary for different subgroups of participants; such issues are 
captured by the box at the bottom left of the figure. Finally, a program is subject to environ- 
mental factors such as employment opportunities, welfare rules, grant levels, and social and cul- 
tural norms. Again, such factors may be significant to policymakers and program developers; a 
program that succeeds in a strong labor market may be ineffective during a recession, and a pro- 
gram that operates one way in a small-town setting like Grand Rapids may perform differently in 
a metropolis like Atlanta. 

Figure 5.2 

Relationship Between Program Assignment and Impacts 



Programmatic 




Environmental 


Factors 




Factors 




Individual 

Characteristics 



The primary analytical problem confronted in this chapter is the fact that variation in 
these different sets of moderating factors is often difficult to isolate, especially when the number 
of sites in a study is limited, as is the case here. Thus, it may seem that a site with a strong en- 
forcement policy is more successful than a site lacking such a policy, but at the same time the 
former site may have a stronger economy, lower grant levels, a different welfare population, or a 
different program focus. How do we separate all these factors? 

Our analyses distinguish among multiple factors in three ways. First, we focus on those 
sites where sample members were randomly assigned to different programs. Such random as- 
signment designs produce the best estimates of the effects of variation in program characteristics. 
Relying on these experimental comparisons, we can draw conclusions about the relative effec- 
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tiveness of HCD programs compared with LFA programs. We can also measure the relative 
benefit of using integrated rather than traditional case management (in which welfare-to-work 
activities and income maintenance activities are carried out by separate caseworkers). 

Second, moving away from randomly created program differences, we approach other 
dimensions — such as enforcement, program focus, and sequencing — by comparing effects across 
all 11 programs, holding constant differences in the individual characteristics of the people 
served by each program. Statistically, these analyses are conducted as follows: 

1. The first step is to estimate a pooled regression model, which identifies the 
overall effect on a particular outcome of assignment to any NEWWS program. 

2. The single NEWWS program variable is replaced by 11 program variables, 
each identifying a single program. A simple F-test is conducted to ascertain 
whether this change in the regression model significantly improves the fit of 
the regression. 5 If it does, the program effects are known to be statistically 
significantly different from one another. 

3. The next step is to interact the original NEWWS program variable with a set 
of individual characteristics of sample members in the study. Adding these in- 
teractions to the regression model identifies the extent to which program ef- 
fects are themselves dependent on the characteristics of those being served. To 
the extent that those characteristics vary across the programs being compared, 
this variation might be falsely attributed to programmatic characteristics, 
thereby inflating apparent differences in program effects across the 11 
NEWWS programs. 

4. After accounting for variation in individual characteristics, the 1 1 program ef- 
fects will be less different from one another, and the F-test used to assess the 
extent of variation across them may no longer find statistically significant 
variation in program effects. 

5. All these results are reported, identifying (a) the degree to which NEWWS 
programs vary in their effectiveness and (b) the extent to which such variation 
results from either programmatic differences or individual differences. 

Third, the analysis will attempt to incorporate variation in the labor market and welfare 
environment across the sites. This is done by taking the estimated program effects from the pre- 
vious step (already adjusted for variation in individual characteristics across the sites) and re- 
gressing these estimates on site-level variables capturing unemployment, job growth, welfare 



5 This is a type of Chow test, measuring the statistical significance of imposing a restriction on a set of coeffi- 
cients. In this context, the unrestricted model is the model in which program effects are allowed to vary across the 1 1 
programs. Replacing 1 1 program variables with a single NEWWS variable forces the program effect to be the same 
for all 1 1 programs. This restriction reduces the fit (and explanatory power) of the regression model. The extent of 
the reduction in explanatory power is a measure of the statistical significance of the variation among individual pro- 
gram effects. See Kennedy, 1992, p. 57, for details. 
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grant levels, etc. Such a regression has few degrees of freedom, which means that results will be 
tentative, but they should be informative nonetheless. 

Finally, we end the chapter by exploring how program effects vary across a number of 
different subgroups pooled across the 1 1 programs. Again, we assess the statistical significance 
of differences in impact estimates across the subgroups. 



IVo Sample amd Emvnrommenit 



Table 5.1 describes the sample for the analyses presented in this chapter, including char- 
acteristics for each sample at the seven NEWWS sites. 6 Although not shown in the table, all dif- 
ferences across the sites were statistically significant, and yet the absolute magnitude of the dif- 
ferences often was small. Here is how the sites compared: 

© On average, sample members in Atlanta were older, had more and older chil- 
dren, and were far more likely than sample members in other sites to be living 
in public housing. They also were least likely to be enrolled already in an edu- 
cation or training activity when they were randomly assigned to the Positive 
Employment and Community Help (PEACH) program (Georgia’s version of 
the JOBS program). 

© Sample members in Grand Rapids were most likely to be enrolled in educa- 
tion and training activities, both at the time of random assignment and in the 
preceding year, when 45.0 percent were enrolled in such an activity. These 
sample members also were relatively young (26.8 years old on average), and 
many had young children. Almost 50 percent of sample members in Grand 
Rapids had a child age 2 or less at the time of random assignment. 

© Most Hispanic sample members in the study lived in Riverside. Also, in this 
site, fewer sample members were never married (about two-thirds were mar- 
ried, separated, divorced, or widowed). Sample members in Riverside also 
were older, had older children, and had more recent work experience than 
sample members in most other sites. The percentage of sample members who, 
as a child, were in a household receiving AFDC was lower than in any of the 
other sites. 

© Sample members in Columbus were least likely to be employed at random as- 
signment or to have ever worked full time for six months or more for one em- 
ployer. They also had long stays on welfare, and they were least likely to have 
received education or training in the year preceding random assignment. 

© In Detroit, more sample members had never been married than in any of the 
other sites. Welfare stays in Detroit were long, and almost half of all sample 
members had been in a household receiving AFDC when they grew up. Few- 
est sample members in Detroit had any earnings in the year preceding random 

6 For more details on the samples and data sources used throughout this report, please refer to Appendix A. 



National Evaluation of Welfare-to-Work Strategies 
Table 5.1 



For Sample Members Without a High School Diploma or GED at Random Assignment: 
Selected Characteristics of Sample Members, by Site 



Characteristic 


Atlanta 


Grand 

Rapids 


Riverside 


Columbus 


Detroit 


Oklahoma 

City 


. Portland 


Demographic Characteristics 
Gender (%) 


Male 


3.6 


4.1 


9.9 


6.4 


3.6 


5.2 


5.7 


Female 


96.4 


95.9 


90.1 


93.6 


96.4 


94.8 


94.3 


Age (%) 


Less than 19 


0.0 


10.7 


0.8 


0.3 


4.8 


19.9 


0.0 


19-24 


10.1 


36.9 


15.8 


13.0 


28.9 


31.9 


24.6 


25-34 


52.9 


36.8 


50.0 


57.0 


40.8 


34.3 


56.7 


35-44 


29.9 


12.4 


25.9 


24.5 


19.7 


11.6 


16.8 


45 and over 


7.2 


3.3 


7.5 


5.1 


5.8 


2.3 


2.0 


Average age (years) 


33.1 


26.8 


32.1 


31.8 


29.5 


25.8 


29.3 


Ethnicity (%) 


Black 


93.0 


39.0 


16.3 


44.4 


85.3 


21.6 


19.8 


White 


4.8 


47.0 


39.2 


53.5 


12.5 


64.3 


68.6 


Hispanic 


1.2 


11.8 


39.4 


0.5 


0.7 


6.0 


5.1 


Other 


0.9 


1.1 


3.9 


1.6 


1.1 


0.8 


3.3 


Family Status 
Marital status (%) 


Never married 


62.1 


63.4 


34.5 


49.6 


69.7 


41.2 


52.1 


Married, living with spouse 


1.6 


3.6 


9.1 


9.4 


3.1 


4.4 


1.6 


Separated 


21.0 


18.4 


32.7 


22.7 


14.9 


35.1 


22.2 


Divorced 


13.7 


13.6 


21.8 


17.3 


10.8 


18.4 


23.5 


Widowed 


1.7 


1.0 


2.0 


1.0 


1.5 


0.9 


0.7 


Number of children (%) 


1 


32.1 


45.4 


35.6 


35.0 


38.4 


54.1 


32.9 


2 


30.6 


35.2 


30.8 


31.9 


28.9 


27.9 


33.2 


3 or more 


37.4 


19.4 


33.6 


33.1 


32.7 


18.0 


33.9 


Age of youngest child (%) 
2 or under 


0.4 


49.9 


7.0 


1.4 


41.0 


51.3 


43.5 


3 to 5 


38.6 


20.5 


49.9 


43.9 


24.0 


20.2 


27.8 


6 or over 


61.0 


29.5 


43.2 


54.7 


35.0 


28.4 


28.7 


Had a child as a teenager (%) 


49.6 


61.4 


36.7 


45.2 


55.5 


63.5 


42.8 
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Table 5.1 (continued) 







Grand 








Oklahoma 




Characteristic 


Atlanta 


Rapids 


Riverside 


Columbus 


Detroit 


City 


Portland 


Labor Force Status 
















Worked full time for 6 months or more 
for one employer (%) 


61.9 


49.8 


62.9 


32.3 


34.3 


52.0 


66.5 


Any earnings in past 1 2 months (%) 


21.0 


37.7 


34.0 


19.5 


15.1 


59.1 


31.3 


Currently employed (%) 


5.1 


8.6 


7.5 


2.8 


5.1 


6.6 


7.2 


Education Status 
















Highest grade completed 
in school (average) 


10.0 


10.0 


10.1 


10.0 


10.2 


9.9 


10.0 


Enrolled in education or training in past 
12 months (%) 


10.2 


45.0 


15.5 


8.9 


20.2 


23.5 


15.2 


Currently enrolled in education 
or training (%) 


5.8 


36.8 


9.7 


8.1 


28.6 


12.0 


8.4 


Public Assistance Status 
















Total prior AFDC receipt (%) a 
















None 


0.4 


0.1 


0.9 


6.9 


2.9 


47.1 


1.1 


Less than 1 year 


15.9 


20.0 


29.9 


6.1 


12.3 


18.3 


16.9 


1 year or more but less than 2 years 


6.8 


18.1 


10.2 


6.3 


7.8 


11.0 


13.9 


2 years or more but less than 5 years 


21.0 


30.2 


26.3 


25.2 


21.3 


14.0 


32.5 


5 years or more but less than 10 years 


22.3 


16.7 


17.3 


26.3 


23.6 


6.8 


24.4 


10 years or more 


33.6 


15.0 


15.5 


29.2 


32.0 


2.7 


11.2 


Raised as a child in a household 
receiving AFDC (%) 


32.2 


41.8 


21.9 


34.0 


48.1 


28.8 


33.2 


Housing Status 
















Current housing status (%) 
















Public housing 


42.2 


2.8 


3.0 


15.2 


5.7 


5.1 


8.8 


Subsidized housing 


21.8 


10.2 


7.7 


23.9 


1.4 


5.5 


15.2 


Emergency or temporary housing 


1.2 


2.2 


1.1 


1.6 


0.8 


16.1 


3.1 


None of the above 


34.8 


84.9 


88.1 


59.3 


92.1 


73.3 


72.8 


Sample size 


2,248 


1,862 


4,695 


3,073 


1,925 


2,585 


1,861 


(continued) 
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Table 5.1 (continued) 



SOURCE: MDRC calculations from information routinely collected by welfare staff 

NOTES: About a quarter of the sample in Riverside’s education-focused program did have a high school diploma or GED, 
but they scored low on the math or reading portion of the appraisal test or had limited English, and thus were determined to 
be in need of basic education. 

Each site was weighted equally in the pooled estimates. 

Distributions may not add up to 100 percent because of rounding. 

All differences across the sites were statistically significant. 

a This refers to the total number of months accumulated from at least one spell on an individual’s own or spouse's AFDC 
case. It does not include AFDC receipt under a parent's name. 



assignment. On the other hand, 28.6 percent already were enrolled in educa- 
tion or training at random assignment — second only to Grand Rapids, the 
other site in Michigan. 

o Oklahoma City had the youngest sample of the seven sites. More than half the 
sample in this site were under 25, and 63.5 percent had a child as a teenager. 
However, in other ways members of the Oklahoma sample were less 
disadvantaged than participants in the other sites: 59.1 percent had worked in 
the previous year, and more than half had been on welfare for less than a year. 

o The ethnic composition of the sample in Portland was similar to that in Okla- 
homa, with a majority of sample members who were white. Sample members 
in Portland also were most likely to have worked full time for six months for 
one employer. 

Thus, looking across these sites, there is a great deal of variation, both in sample mem- 
bers’ work-readiness and in their current and recent participation in education and training. One 
might expect all this to be reflected in the program effects found in these sites, although the ex- 
pected pattern of effects is not obvious. For example, in sites like Atlanta, Riverside, and Port- 
land few sample members had recent experience with education and training. In those sites, one 
might expect the programs to make a bigger difference in that regard than in sites such as Grand 
Rapids and Detroit, where many sample members already participated on their own. Another im- 
portant factor might be the long-term welfare dependency of sample members across the sites. 
On the one side, we find Oklahoma City, where only about 9.5 percent of the sample had re- 
ceived AFDC for five years or more. On the other side, we find sites like Columbus, Detroit, and 
Atlanta, in which around 55 percent received AFDC that long. One might expect program effects 
to be larger in the latter group of sites, where control group members would be less likely to 
leave AFDC on their own. On the other hand, however, samples of long-term recipients tend to 
be more disadvantaged overall, making it more difficult to serve them. In any case, it seems im- 
portant to consider variation in sample characteristics as we attempt to explain variation in pro- 
gram effects across the sites. In later sections, we will use statistical controls to do so, and we 
also will conduct subgroup analyses to uncover how some of these characteristics interact with 
the programs across the different sites. 

Table 5.2 describes the sites in a different way, using aggregate statistics covering the 
economic and welfare environment. Although these statistics do not always cover the same cal- 
endar years, they give a useful overview of the environment. Looking at the table, it appears that, 
in 1997, unemployment was below 5 percent in all sites except Riverside. Between 1996 and 
1997, unemployment fell in all the sites except Oklahoma City, and this reduction was strongest 
in Grand Rapids. Employment growth, measured between 1990 and 1996, was strongest in At- 
lanta, Grand Rapids, and Portland, but it was generally strong in all the sites. 

Poverty rates, estimated by the Census Bureau for 1995 only, varied substantially across 
the sites. Highest in Atlanta and Detroit, they ranged from highs of 20.9 and 20.6 percent in those 
two sites to a low of 9.6 percent in Grand Rapids. This means that sample members in the latter 
site (and in comparable places such as Portland and Columbus) live in communities that may of- 
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